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1 .0  List  of  objectives  of  the  research  effort 

1.1  How  do  we  ensure  the  optimum  coupling 
of  the  information  medium  to  the  human? 

1 .2  Are  there  context  and  task  impacts  on 
perceptual  requirements? 

1 .3  How  should  we  combine  information 
elements  into  a  "world"  construct  which  rapidly 
communicates  situation  awareness? 

1 .4  What  are  the  combinatorial  rules  which 
optimize  this  communication? 

1 .5  What  is  the  relation  between  "presence" 
and  display  effectiveness? 

1 .6  How  do  we  measure  situation  awareness? 

1 .7  What  are  the  components  of  situation 
awareness? 

1 .8  How  do  we  measure  spatial  awareness? 

1 .9  What  is  the  use  of  mental  models?  Why 
compare  them  with  real-time  information  input? 

1.10  Can  we  dynamically  adapt  the  interface  to 
match  the  user? 


Abstract 


The  report  documents  the  first  year  of  work  on  a  four-year  project  titled 
“Communicating  situation  awareness  in  virtual  environments.”  Included  in  the 
report  are  references  to  the  fifteen  papers  that  were  produced,  and  descriptions 
of  eleven  of  the  research  projects  that  were  started.  In  addition,  there  is  a 
description  of  a  workshop  on  virtual  reality  which  was  hosted  at  the  University  of 
Washington,  and  which  was  attended  by  10  federal  labs.  Other  work  performed 
during  the  period  and  described  in  the  report  includes  the  conceptual  and 
software  development  of  a  virtual  world  {the  “Towering  Inferno")  for  performing 
experimental  manipulations,  and  the  detailed  design  of  a  virtual  reality  testbed. 
As  a  part  of  the  infrastructure  for  this  line  of  research,  a  knowledge  base  was 
also  developed.  This  knowledge  base  is  structured  to  be  compatible  with 
ongoing  nationwide  efforts  for  electronic  storage  and  retrieval  of  information. 
Ten  objectives  of  the  research  effort  are  detailed.  The  report  provides 
substantive  evidence  that  the  project  is  on  schedule,  and  making  effective  use 
of  the  available  facilities  and  support. 


1.0  List  of  objectives  of  the  research  effort 


Over  the  period  of  the  grant,  it  is  our  objective  to  address  a  number  of  questions 
which  are  fundamental  to  communicating  situation  awareness  in  virtual 
environments.  These  are  listed  below  as  bulleted  items,  and  are  treated  in 
more  detail  in  later  paragraphs. 

•  How  do  we  ensure  the  optimum  coupling  of  the  information  medium 
to  the  human? 

•  Are  there  context  and  task  impacts  on  perceptual  requirements? 

•  How  should  we  combine  information  elements  into  a  "world"  construct 
which  rapidly  communicates  situation  awareness? 

•  What  are  the  combinatorial  rules  which  optimize  this  communication? 

•  What  is  the  relation  between  "presence"  and  display  effectiveness? 

•  How  do  we  measure  situation  awareness? 

•  What  are  the  components  of  situation  awareness? 

•  How  do  we  measure  spatial  awareness? 

•  What  is  the  use  of  mental  models?  Why  compare  them  with  real-time 
information  input? 

•  Can  we  dynamically  adapt  the  interface  to  match  the  user? 

1.1  How  do  we  ensure  the  optimum  coupling  of  the  information 
medium  to  the  human? 

It  is  necessary  and  important  to  make  a  distinction  between  the  medium  and  the 
message.  The  medium  refers  to  the  virtual  interface  hardware  which  is  used  to 
convey  appropriate  visual  and  acoustic  stimuli  to  the  human  to  create  a  virtual 
environment.  The  message  refers  to  organization  of  the  informa;(on  elements, 
and  their  embodiment  into  a  world  construct  or  archetype.  One  of  our  objectives 
is  to  insure  that,  at  a  signal  level,  we  achieve  optimum  coupling  of  the 
information  medium  to  the  human.  This  will  require  investigating  elements  such 
as  image  quality,  image  stability,  and  the  spatial  and  temporal  coherency  of 
multi-modal  presentations,  to  determine  both  the  minimum  and  optimum 
requirements  for  coupling. 


1 .2  Are  there  context  and  task  impacts  on  perceptual  requirements? 

There  are  stories  of  pilots  who,  in  the  heat  of  battle,  are  impervious  to  a  range  of 
visual  and  auditory  stimuli.  This  can  have  major  detrimental  effects  on  situation 
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awareness  and  survivability.  It  remains  to  be  investigated  whether  this  type  of 
perceptual  gain  control  is  context  and  task  specific,  and  whether  there  are 
techniques  for  reopening  the  channel  of  communication  under  these  conditions. 


1 .3  How  should  we  combine  information  elements  into  a  "world" 
construct  which  rapidly  communicates  situational  awareness? 

It  is  our  contention  that  the  formulation,  by  the  operator,  of  a  world  construct  or 
archetype,  is  essential  for  the  development  of  situational  awareness. 
Furthermore,  it  is  our  hypothesis  that  the  manner  in  which  these  elements  are 
combined  and  presented  may  affect  the  formulation  of  this  archetype.  We 
intend  to  test  this  hypothesis. 


1 .4  What  are  the  combinatorial  rules  which  optimize  this 
communication? 

By  performing  experimental  investigations  into  the  way  in  which  information 
elements  are  combined,  we  propose  to  formulate  design  guidance  for 
optimizing  the  effects  of  these  combinations.  It  is  planned  that  these 
combinatorial  rules  will  provide  guidance  for  multi-sensory  inputs,  as  well  as  for 
combinations  of  elements  in  one  modality. 


1 .5  What  is  the  relation  between  "presence"  and  display 
effectiveness? 

Presence  maybe  a  product  of  the  imagination,  it  is  the  experience  of  being 
there,  or  conversely,  of  something  being  here.  The  experience  can  be  aided  by 
suitable  props,  including  displays,  anJ  other  VR  technology.  The  interest  in 
presence  stems  from  the  contention  that  with  it,  there  are  certain  performance 
enhancements.  Therefore,  a  starting  point  for  work  in  this  area  must  be  a  test  of 
this  hypothesis.  Other  questions  which  will  be  addressed  include:  Can 
presence  be  operationally  defined?  What  are  the  prerequisites  for  presence? 
How  can  presence  be  measured?  What  are  the  contributions  of  the  medium 
and  the  message? 


1 .6  How  do  we  measure  situation  awareness? 

Situation  awareness  has  been  defined  with  minor  variations  by  many  authors, 
including  Sarter  and  Woods  (1991),  Endsley  (1988),  Fracker  (1991),  Tolk  and 
Keether  (1982),  McKinnon  (1986),  and  Whitaker  and  Klein  (1988).  A 
consensus  definition  would  be  that  situation  awareness  refers  to  an 
understanding  of  the  current  state  of  the  subject’s  environment,  together  with  an 
ability  to  predict  changes  to  the  environment. 
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Techniques  for  measuring  it  include  having  subjects  respond  to  questions 
about  the  state  of  their  environment  (Endsley,  1987,  1988a),  and  looking  at 
responses  to  specific  events  (Fracker,  1991).  Other  authors  have  explored  the 
use  of  a  techniques  which  assess  the  dynamic  aspects  of  situational  awareness 
(Barter  and  Woods,  1991).  We  intend  to  use  a  combination  of  the  techniques, 
and  others,  to  address  the  issue. 


Endsley, M.R.  “Design  and  Evaluation  for  Situation  Awareness  Enhancement", 
Proceedings  of  the  Human  Factors  Society  -  32nd  Annual  Meeting  --  1988. 

Endsley,  M.R.  "SAGAT;  A  Methodology  for  the  Measurement  of  Situation 
Awareness,"  Northrop  Technical  Report:  NOR  DOC  87-83,  August  1987. 

Endsley,  M.R.  "The  Functioning  and  Evaluation  of  Pilot  Situation  Awareness," 
Northrop  Technical  Report:  NOR  DOC  88-30,  April  1 988a. 

Fracker,  M.L  Measures  of  Situation  Awareness:  An  Experimental  Evaluation 
(U)",  AL-TR-1991-0127. 

Tolk,  J.D.,  and  Keether,  G.A.  (1982).  Advanced  medium-range  air-to-air  missile 
(AMRAAM)  operational  evaluation  (OUE)  final  report  (U).  Air  Force  Test  and 
Evaluation  Center,  Kirtland  Air  Force  Base,  NM. 

McKinnon,  F.B.  (ed.).  (1986).  Final  report  of  the  intraflight  command,  control, 
and  communications  symposium  (U).  57th  Fighter  Weapons  Wing,  Nellis  Air 
Force  Base,  NV. 

Barter,  N.B.,  and  Woods,  D.D.,  (1991).  Situation  awareness:  A  critical  but  ill- 
defined  phenomenon.  The  International  Journal  of  Aviation  Psychology,  1(1), 
45-57. 

Whitaker,  L.A.,  and  Klein,  G.A.  (1988).  Situation  awareness  in  the  virtual  world: 
Situation  assessment  report.  Proceedings  of  the  11th  Symposium  of 
Psychology  in  the  Department  of  Defense,  April. 


1 .7  What  are  the  components  of  situation  awareness? 

In  our  proposal  to  the  AFOSR  we  proposed  a  model  of  situation  awareness 
which  included  the  components  of:  spatial  awareness  factors  (geometry),  state 
awareness  factors  (status),  and  situation  awareness  factors  (problem  solving). 
It  is  our  intention  to  refine  this  model  further  and  to  explore  some  of  the 
components,  and  their  relative  contributions  to  situation  awareness. 
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1 .8  How  do  we  measure  spatial  awareness? 


Spatial  Awareness  is  a  major  component  of  Situational  Awareness.  Therefore, 
any  interface  which  strives  to  communicate  situational  awareness,  must  also 
communicate  spatial  awareness  (Endsley,  1988).  Spatial  awareness  is 
manifested  as  the  ability  to  move  through  an  environment  efficiently  and  locate 
items  or  areas.  Attaining  spatial  awareness  allows  one  to  make  transformations 
on  mental  representations  of  an  area  so  as  to  view  and  recognize  it  from 
different  perspectives.  The  components  involved  in  spatial  awareness  are 
spatial  orientation,  spatial  visualization  and  understanding  spatial  relationships 
(Lohman,  1979).  Spatial  orientation  also  allows  mental  manipulation  of  an 
object  using  oneself  for  reference.  Spatial  visualization  goes  further  in  that  the 
person  can  manipulate  the  relationships  within  an  object  (Newcombe,  1982). 
The  third  component  involves  understanding  the  spatial  relationships  between 
oneself  and  the  environment,  and  between  two  objects.  The  perceptions  are 
influenced  by  displays  of  the  world,  field  of  view  and  the  amount  of  experience 
in  the  environment  (Andre,  Wickens,  Moorman  &  Boschelli,  1991).  Experience 
can  be  of  an  egocentric  and/or  exocentric  manner. 

There  are  a  number  of  methods  for  measuring  spatial  awareness.  These 
include  target  replacement  metrics  (Wells,  Venturino  &  Osgood,  1988),  cued 
target  position  discrimination  (Palmer,  1990),  distance  estimation  (Euclidean 
and  shortest  path)  between  two  landmarks,  and  directional  pointing  (Sholl, 
1987),  Also,  there  are  metrics  from  the  domain  of  ecological  psychology  which 
may  be  of  benefit.  We  intend  to  explore  these  metrics,  and  develop  our  own. 


References: 

Andre,  A.D.,  Wickens,  C.D.,  Moorman,  L,  &  Boschelli,  M.M.  (1991).  Display 
formatting  techniques  for  improving  situation  awareness  in  the  aircraft  cockpit. 
International  Journal  of  Aviation  Psychology. 

Endsley,  M.R.,  (1988).  Design  and  evaluation  for  situation  awareness 
enhancement.  Proceedings  of  the  Human  Factors  Society  -32nd  Annual 
Meeting. 

Lohman,  D.F.  (1979).  Spatial  Ability:  Review  and  re-analysis  of  the 
correlational  literature,  Stanford  University  Technical  Report  8.  Cited  in  Infield, 
S.E.  (1991).  An  investigation  into  the  relationship  between  navigation  skill  and 
spatial  abilities.  Doctoral  Dissertation.  University  of  Washington. 

Newcombe,  N.  (1982).  Sex-related  differences  in  spatial  ability:  Problems  and 
gaps  in  current  approaches.  In  Potega,  M.  (Ed.)  Spatial  Abiliti^  Development 
and  Physiological  Foundations.  223-250.  New  York;  Academic  Press. 

Palmer,  J.  (1990).  Attentional  limits  on  the  perception  and  memory  of  visual 
information.  Journal  of  Experimental  Psychology:  Human  Perception  and 
Performance,  16,  332-350. 
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Shell,  M.J.  (1987).  Cognitive  maps  as  orienting  schemata.  Cognition,  4, 
615-628. 

Wells,  M.J.,  Venturino,  M.,  &  Osgood,  R.K.  (1989).  The  effect  of  field-of-view  size 
on  performance  at  a  simple  simulator  air-to-air  mission.  Proceedings  of  SPIE, 
Helmet-Mounted  Display.,  1116 , 1 26-1 37. 


1 .9  What  is  the  use  of  mental  models?  Why  compare  them  with  real¬ 
time  information  input? 

A  mental  model  is  the  organization  of  knowledge  about  the  environment,  a 
specific  system,  or  other  things  with  which  people  interact  (Norman,  1988).  It 
forms  the  basis  for  understanding  a  system  and  predicting  its  future  behavior 
(Wickens,1992).  People  form  mental  models  through  experiences,  training,  or 
instructions. 

Mental  models  are  not  always  correct  representations  of  systems  or 
environments.  However,  when  they  are  correct,  mental  models  can  contribute 
to  complex  problem  solving.  It  has  been  found  that  a  person  who  learns  a 
system  operation  with  the  use  of  a  mental  model  is  able  to  solve  a  novel 
problem,  whereas  those  who  learned  with  procedural  instructions  are  not  able 
to  do  so  (Wickens,  1992).  An  accurate  mental  model  of  a  system  can  be 
beneficial  because  it  provides  the  user  with  knowledge  that  is  useful  when  other 
leaned  procedures  fail.  We  are  Interested  In  Investigating  mental  models 
because  of  their  impact  on  situation  awareness. 

In  using  virtual  reality  technology  to  improve  situation  awareness,  it  is 
necessary  to  consider  the  relationship  between  mental  models  and  information 
inputs.  Real-time  information  represents  the  dynamic  state  of  some  parts  of  the 
system.  There  are  three  representations  which  must  be  considered  when  an 
interface  is  designed  (Wickens,  1992): 

1 )  The  physical  system 

2)  The  internal  representation 

3)  The  interface  between  the  above  two 

It  is  important  to  maintain  a  high  degree  of  compatibility  among  these  three 
representations  so  that  information  inputs  can  be  compatible  with  the  operator’s 
Information  processing  needs,  or  mental  models  of  the  system.  Without  such 
compatibility,  longer  information  processing  is  required,  or  in  the  worst  case, 
errors  result.  In  a  complex  and  dynamic  environment,  it  is  especially  crucial  to 
have  compatibility  between  information  inputs  and  mental  models  to  allow 
accurate  and  quick  responses  from  users. 


Norman,  D.A.  (1988).  The  Psychology  of  Everyday  Things.  New  York:  Harper  & 
Row. 
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Wickens,  C.D.  (1992).  Engineering  Psychology  and  Human  Performance  (2nd 
Ed.).  Harper  Collins  Publishers. 


1.10  Can  we  dynamically  adapt  the  interface  to  match  the  user? 

One  of  the  attributes  of  an  ideal  interface  would  be  the  ability  to  dynamically 
adapt  to  the  user.  Such  an  interface  may  make  it  possible,  for  example,  to 
provide  the  optimum  clustering  of  state  and  spatial  information  during  a  mission 
segment  which  required  the  user  to  switch  from  an  offensive  to  a  defensive 
mode  of  operation.  In  order  for  this  to  occur,  it  is  necessary  to  measure  the 
user’s  requirements,  unobtrusively  if  possible,  as  well  know  what  to  do  with  this 
information.  We  intend  to  investigate  techniques  for  measuring  user  state.  In 
combination  with  the  knowledge  gained  from  our  explorations  into  display 
effectiveness,  and  coupling  of  the  human  to  the  medium,  we  intend  to  develop 
methods  of  matching  the  interface  to  the  user. 


2.0  Status  of  the  research  effort. 

2.1  Research  efforts  underway 

The  AFOSR  funding  is  currently  being  used  to  support  i  i  research  studies.  The 
titles  of  these  efforts  are  listed  below,  and  descriptions  of  the  goals  and 
research  plans  are  included  in  subsequent  sections. 

2.1 .1  List  of  research  efforts 

•  Relationship  Between  Presence  and  Attentional 
Resources  in  Virtual  Environments. 

•  Increasing  Spatial  Awareness  in  Exocentric  Displays 

•  Effect  of  a  Virtual  Body  on  Spatial  Awareness 

•  Effects  of  Spatialized  Sound  on  Presence  and 
Performance  Within  Virtual  Environments 

•  Facial  Expression  Exhibition  in  Human  Computer 
Interaction 

•  Subjective  Measures  of  Presence 

•  Memory  for  virtual  experiences 

•  Virtual  Reality  Monitoring 

•  Measuring  Presence  in  Terms  of  Subjective  Reference 
Frames 

•  Navigation  and  Wayfinding  in  VR;  Finding  the  Proper 
Navigational  Tools  and  '^ues  to  Build  Navigational 
Awareness 

•  Effective  Information  Presentation  to  Facilitate  Decision 
Making  Under  Uncertainty 
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2.1.2  Relationship  Between  Presence  and  Attentional  Resources 
in  Virtual  Environments. 

Ove  L.  Bjorneseth  (supervised  by  Woody  Barfield) 

The  purpose  of  the  research  is  to  discover  how  combinations  of  different 
sensory  modalities  compete  for  attentional  resources  within  the  real  world  and 
virtual  environments  and  how  presence  is  influenced  by  the  allocation  of 
attentional  resources. 

It  is  well  known  that  people  have  difficulties  in  performing  multiple  tasks  at  the 
same  time.  This  arises  from  the  fact  that  people  have  a  limited  amount  of 
attentional  resources.  In  other  words,  people  can  pay  attention  to  only  so  much 
information  at  the  same  time.  The  concept  of  workload  is  well  investigated  and 
closely  tied  to  attentional  resources,  it  is  reasonable  to  assume  that  a  similar 
relationship  exists  between  workload  and  presence.  That  is,  as  attentional 
resources  are  allocated  more-and-more  to  virtual  stimuli,  presence  for  virtual 
stimuli  should  increase. 

Independent  variables  in  this  research  will  focus  on  different  types  of  sensory 
information  (visual,  auditory,  tactile  and  kinesthetic),  together  with  display 
variables  (FOV,  resolution  and  frame  rate).  Dependent  variables  are  task 
performances  and  subjective  questionnaires  in  addition  to  physical  measures 
(EEG,  pulse,  blood  pressure  and  skin  conductivity). 

Currently,  a  total  immersive  testbed  is  under  development  in  the  Laboratory  for 
Interactive  Computer  Graphics  and  Human  Factors  that  will  make  it  possible  to 
move  stimuli  between  the  real  and  the  virtual  environment. 


2.1.3  Increasing  Spatial  Awareness  in  Exocentric  Displays 
Gina  B.  Crvarich  (supervised  by  Suzanne  Weghorst) 

Introduction 

Spatial  awareness  is  a  key  component  of  situation  awareness.  Within  virtual 
environments,  it  is  generally  argued  that  as  the  fidelity  of  the  sensory  input  to  a 
participant  increases,  the  sense  of  presence  and  possibly  of  situation 
awareness  will  also  increase.  This  project  will  evaluate  how  virtual 
environments  can  increase  one's  spatial  and  thus  situation  awareness. 

Goal:  Maximize  the  perceptual  fidelity  of  3D  spatial  displays  by  manipulating  a 
range  of  computer  generated  depth  cues. 

From  an  Exocentric  Point  of  View 

A  controller  is  an  outside  participant  in  a  specific  mission  who  needs  to  rapidly 
assess  a  critical  situation  and  then  communicate  information  to  an  inside 
participant.  It  has  been  reported  that  an  exocentric  display  (god's-eye-view 
frame  of  reference)  is  superior  to  an  egocentric  display  for  developing  an 
accurate  mental  model  of  a  flight  environment  formed  during  a  mission.  This 
project  will  use  an  exocentric  display  to  facilitate  a  controller's  spatial 
understanding  of  a  mission  environment. 

Computer  Graphics  Displays 

In  the  real  world,  we  use  a  variety  of  depth  cues  to  recognize  and  visualize, 
relative  spatial  locations  of  objects.  Real  world  depth  cues  can  be  artificially 
created  via  specific  characteristics  of  computer  graphic  displays.  The  following 
is  a  partial  list  of  depth  cues  and  how  they  are  recreated  within  computer 
graphic  systems; 

Real  World  Cue 

Monocular  Cues 
linear  perspective 
texture  gradient 
motion  parallax 
Kinetic  Depth  Effect 

Binocular  Cues 
stereopsis  stereoscopic  displays 

The  effective  use  of  computer  graphic  display  technology  is  needed  in  order  to 
create  useful  exocentric  displays.  The  creation  of  each  depth  cue  is 
computationally  expensive,  thus  only  depth  cues  essential  to  increased  spatial 
awareness  should  be  used.  Parameters  of  an  effective  exocentric  display  will 
be  determined  by  varying  the  computer  graphic  cues  listed  above. 

Experimental  Tasks 

The  actual  experimental  tasks  to  measure  a  controller's  spatial  awareness  are 
under  development.  An  area  currently  under  consideration  is: 


Computer  Graphics  Cue 

perspective  projection 
varying  texture  maps 
motion  tracking 
rotational  displays 
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"What  Do  They  See?"  Task 

Goal  Evaluate  the  controller's  ability  to  undfistancl  the  spatial  viewpoint  of  an 
inside  parlicipant  as  well  as  the  controllers  undeistanding  ot  ihe  mission 
environment 

Measure:  Given  an  exocentnc  display  which  includes  an  inside  participant 

with  a  specific  viewing  direction,  the  controller  will  judge  what  the  inside 
participant  would  actually  be  seeing. 


Display  Options 


Stereoscopic  Viewing 

■  Evaluate  performance  with 
monoscopic  vs.  stereoscopic 
viewing 


Rotation 

■  Evaluate  performance  wun 
rotation  around  multiple  axes 

■  Evaluate  performance  with 
monoscopic  vs.  stereoscopic 
viewing 

■  Obsen/e  rotation  strategies 


Head-Motion  Tracking 

■  Evaluate  performance  with  and 
without  rotation 

■  Evaluate  performance  with 
monoscopic  vs.  stereoscopic 
viewing 

■  Observe  head-motion 
strategies 
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2.1.4  Effect  of  a  Virtual  Body  on  Spatial  Awareness 
Mark  H.  Draper  (supervised  by  Max  Wells) 

Introduction 

VR  offers  many  freedoms  regarding  self-representation  as  a  'virtual  body  (VB)': 

-  Size 

-  Shape 

-  Level  of  Realism 

-  Body  Configuration  (from  no  VB  to  a  full  body  part 
representation) 

The  representation  of  self  is  fundamental  to  virtual  interface  design.  The 
tremendous  flexibility  described  above  needs  to  be  harnessed  by  performance 
metrics  to  determine  what  VB  configuration  is  appropriate  for  a  defined  task. 

Design  Question 

"Include  a  VB?" 

"VES" 

-  Increases  level  of  Presence 

-  Self-expression/identity  in  virtual  commons 

"A/0" 

-  Hardware/Software  costs 

-  Cumbersome  body  interface  devices 

-  Performance  benefits  ill-defined 

Thesis  Question: 

Given: 

•  AFOSR  Contract  focuses  on  transmitting  Situation  Awareness  in  VR, 
and  Spatial  awareness  is  assumed  to  be  a  subset  of  Situation  Awareness, 

Does  the  existence  of  a  VB  in  a  virtual  environment  enhance  one's 
spatial  awareness?  If  so,  what  virtual  body  elements  contribute  to  this  effect? 

Definitions: 

-  VB:  The  representation  of  one's  own  body  (or  subset  of  it)  that  can  be 
self  viewed  in  a  virtual  world. 

-  Spatial  Awareness:  An  awareness  of  the  relative  locations  of  objects  in 
the  environment  in  relation  to  the  participant. 

Research  Ideas: 

-  Pilot  Study:  A  quick  look  to  see  if  humans  use  their  own  body  as  a 
contextual  aid  in  absolute  distance  estimation  tasks  in  the  real  world. 

-  Main  Study:  A  carryover  of  the  pilot  study  into  VR,  where  subjects  will 
perform  spatial  awareness  tasks  and  the  VB  can  be  manipulated  freely.  Again 
the  emphasis  will  be  on  observing  any  spatial  awareness  enhancements  that 
can  be  associated  with  the  existence  of  a  VB. 
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Potential  VB  Configurations 

-  Arms  Only 

-  Full  VB 

-  No  VB 

-  Static/Limited  Dynamics 

-  Other? 


2.1.5  Effects  of  Spatialized  Sound  on  Presence  and  Performance 
Within  Virtual  Environments 
Robert  G.  Futamura  (supervised  by  Max  Wells) 

In  many  cases,  the  presentation  of  spatialized  auditory  cues  in  addition  to  visual 
cues  may  increase  performance  measures  on  certain  tasks.  In  addition,  the 
sense  of  presence  may  be  desired  in  order  to  facilitate  some  performances 
within  a  virtual  environment.  The  veridical  (re)construction  of  sounds  within  a 
virtual  environment  may  increase  the  likelihood  of  conveying  a  sense  of 
presence,  or  being  within  the  virtual  environment. 

Presence  is  commonly  defined  to  result  from  the  match  of  input  and  output 
channels  of  both  the  human  operator  and  the  virtual  interface.  With  the  addition 
of  a  realistic  3D  auditory  channel  to  the  existing  virtual  interface,  it  is  expected 
that  levels  of  performance  and  presence  would  increase. 

Several  factors  to  manipulate  include; 

•  spatial  vs.  non-spatial  vs.  no  sound  sources 

•  static  vs.  dynamic  sound  sources 

•  interactive  vs.  non-interactive  sources 

•  number  of  sound  sources 
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2.1.6  Subjective  Measures  of  Presence 
Claudia  Hendrix  (supervised  by  Woody  Barfield) 

Claudia’s  research  effort  is  twofold:  (1)  the  development  of  metrics  for  the 
evaluation  of  presence  and  performance  within  virtual  environments,  and  (2)  an 
investigation  of  the  resolution  of  computer-generated  sensory  input  necessary 
for  a  person  to  "get  a  feeling"  of  presence  within  a  virtual  environment.  Factors 
which  she  is  investigating  include  the  visual  field  of  view,  update  rate, 
stereopsis,  headtracking  and  degradation  of  the  sensory  input  (i.e.,  resolution) 
in  terms  of  presence  and  performance  of  navigation  tasks  and  spatial 
discrimination.  The  investigation  will  be  conducted  by  evaluating  the 
performance  of  subjects  in  a  computer-generated  environment  where  both  the 
visual  resolution,  field  of  view,  and  frame  update  rate,  etc.  will  be  varied.  The 
approach  taken  for  the  development  of  metrics  is  a  standardized  subjective 
questionnaire  which  will  relate  variables  used  to  generate  virtual  environments 
with  presence.  Currently  a  literature  search  is  being  conducted  on  hardware 
and  software  variables  which  may  influence  presence  and  also  on  metrics  to 
measure  presence  and  performance  within  virtual  environments.  Exploratory 
studies  on  the  sense  of  presence  within  visual  and  auditory  virtual 
environments  as  well  as  in  spatial  discrimination  have  been  conducted. 
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2.1.7  Memory  for  virtual  experiences 
Hunter  G.  Hoffman 


Abstract 

We  investigated  whether  subjects  could  separate  memories  of  events 
experienced  in  virtual  reality  from  real  and  imagined  events;  Virtual-reality 
monitoring.  Subjects  studied  8  separate  spatial  configurations  of  real  geometric 
objects  arranged  on  a  life-sized  chessboard,  8  configurations  in  virtual  reality 
(an  immersive,  computer-simulated  world),  and  imagined  objects  in  8  other 
configurations.  On  a  later  identification  of  origin  memory  test,  subjects  were 
generally  able  to  correctly  identify  the  sources  of  the  events.  Results  are 
interpreted  within  the  Johnson-Raye  (1981)  theoretical  framework. 

According  to  a  number  of  researchers  (e.g.,  Barfield,  Slater,  Zeltzer,  and 
Sheridan,  in  press),  researchers  presently  lack  a  theoretical  framework  within 
which  to  study  virtual  reality,  and  lack  an  objective  measure  of  "presence",  the 
feeling  that  one  is  "in  a  place"  when  immersed  in  a  virtual  environment.  The 
goal  of  our  research  is  to  develop  a  theoretical  framework  to  guide  research  on 
virtual  reality,  and  to  develop  a  measure  of  presence. 

Reality  monitoring  is  the  decision  process  by  which  memories  of  real  and 
imagined  events  are  distinguished.  According  to  Johnson  and  colleagues  (see 
Johnson,  Hastroudi,  &  Lindsay,  1993,  for  a  review),  differences  between  real 
and  imagined  events  as  originally  experienced  are  preserved  in  memory  and 
can  later  serve  as  cues  to  where  the  memory  originated.  That  is,  memoiy 
source  is  inferred  by  the  subject  at  the  time  of  retrieval,  based  on  cues 
associated  with  the  target  memory.  Reality  monitoring  decisions  take 
advantage  of  differences  in  qualitative  characteristics  of  memories  from  different 
sources.  Examples  of  reality  monitoring  decisions  in  the  real  world  include 
separating  memories  of  what  one  actually  witnessed  in  a  crime  from  imagined 
"elaborations";  separating  fact  from  fantasy  with  respect  to  childhood  sexual 
encounters;  and  unconscious  plagiarism  (unwittingly  identifying  the  origin  of  an 
idea  as  original  when  in  fact  someone  else  told  you  the  idea). 

In  research  just  completed,  we  investigated  whether  subjects  could  discriminate 
memories  of  events  experienced  in  virtual  reality  from  real  and  imagined 
events.  Subjects  studied  the  spatial  locations  of  geometric  objects  located  on  a 
life-sized  chessboard.  The  objects  were  either  real,  imagined,  or  virtual 
(presented  in  a  computer  simulated  reality).  Subjects  later  took  a  memory  test 
requiring  them  to  identify  the  source  of  each  spatial  configuration.  They  were 
shown  a  number  of  2-d  diagrams  of  spatial  configurations  of  geometric  objects 
they  had  seen  in  the  study  phase  (and  distracters).  For  each  test  item,  they 
were  to  choose  whether  they  had  seen  it  in  the  real  world,  had  imagined  the 
objects  based  on  instructions,  had  seen  that  configuration  of  objects  in  virtual 
reality,  or  if  it  was  new  (not  in  the  study  phase).  Following  the  logic  used  by 
Johnson  and  colleagues,  we  predicted  that  some  of  the  differences  in  the 
qualities  of  the  experiences  as  originally  experienced  would  get  stored  in 
association  with  the  target  memories,  and  would  allow  subjects  to  infer  the 
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sources  of  their  memories  on  a  later  memory  test.  More  specifically,  we 
predicted  that  cues  associated  with  memories  of  virtual  events  would  allow  the 
virtual  source  of  these  memories  to  be  identified.  As  predicted,  we  found  that 
subjects  were  generally  able  to  discriminate  memories  of  virtual  events  from 
memories  of  real  and  imagined  events,  establishing  the  reality  monitoring 
paradigm  as  a  theoretical  framework  in  which  to  study  similarities  and 
differences  between  real,  virtual  and  imagined  experiences. 

Keith  Hullfish  and  I  are  now  investigating  whether  reality  monitoring 
confessions  can  serve  as  an  objective  measure  of  presence.  Perhaps  virtual- 
reality  monitoring  can  be  used  as  a  metric  for  assessing  how  closely  virtual 
events  simulate  real  events,  a  sort  of  Turing  test  for  quantifying  the  fidelity  of 
virtual  technologies.  The  more  convincing  the  virtual  world,  the  more  "present" 
subjects  will  feel.  Increasing  the  feeling  of  presence  and  the  quality  of  the 
virtual  experience  will  increase  the  similarity  between  the  virtual  and  real 
experiences,  making  subjects  more  likely  to  get  memories  of  virtual  and  real 
events  confused. 
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2.1.8  Virtual  Reality  Monitoring 

Keith  C.  Hullfish  (supervised  by  Max  Wells) 

The  current  research  effort  includes  adapting  a  paradigm  from  cognitive 
psychology,  Source  Monitoring  (Johnson,  Hashtroudi,  &  Lindsay;  1993),  within 
which  to  study  Virtual  Reality.  Virtual  Reality  Monitoring  is  the  decision  process 
by  which  people  distinguish  between  real,  virtual,  and  imagined  origins  of 
memories  (i.e.,  sources).  Historically,  this  paradigm  has  been  used  to  examine 
the  differences  between  external  (real,  virtual)  and  internal  (imaginary)  sources 
of  information  in  memory.  Evidence  of  differences  between  source  events,  as 
originally  experienced,  are  preserved  in  memory  and  can  later  serve  as  cues  to 
where  the  memory  originated.  Monitoring  decisions  are  based  on  the 
qualitative  differences  between  the  memories.  Source  of  a  memory  is  inferred 
based  on  the  average  differences  of  memory  characteristics,  in  combination 
with  judgment  processes.  These  memory  characteristics  are: 
sensory/perceptual  information,  contextual  information  (spatial  and  temporal), 
semantic  detail,  affective  information,  and  cognitive  operations.  Qualities 
typically  associated  with  a  real  event  include  more  sensory/perceptual  and 
contextual  information  whereas  imaginary  events  reflect  mental  effort. 

The  research  effort  is  developing  the  concept  as  it  applies  to  Virtual  Reality  and, 
thus,  will  examine  the  similarities  and  differences  of  virtual  experiences  with 
both  real  and  imaginary  experiences,  it  is  proposed  that  the  limitations  that  the 
technology  impose  on  the  senses,  prevent  a  person  from  fully  contextualizing 
information  at  acquisition,  which  could  reduce  encoding  of  potentially  relevant 
external  source  information.  In  addition  to  degrading  the  sensory/perceptual 
and  contextual  information  available  to  a  participant,  additional  mental  effort 
could  be  required.  According  to  the  theoretical  framework,  these  characteristics 
could  increase  source  monitoring  confusions  between  external  (virtual)  and 
internal  (imaginary)  events  than  otherwise  would  have  resulted  if  the 
technology  simulated  the  real  world  perfectly.  These  characteristics  could  also 
serve  as  a  cue  to  a  virtual  source,  and  thus  distinguish  a  virtual  event  from  real 
and  imagined  events. 

An  initial  study  was  conducted  (Hoffman,  Hullfish,  and  Houston;  in  preparation) 
to  establish  the  validity  of  adopting  this  paradigm.  Subjects  were  able  to 
distinguish  between  real,  virtual,  and  imagined  events,  indicating  that  virtual 
experiences  have  unique  memory  characteristics.  Efforts  are  underway  to 
refine  the  experimental  procedure  and  developing  the  concept  of  Virtual  Reality 
Monitoring  confusion  scores  as  a  metric  for  presence  in  virtual  environments. 
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2.1.9  Facial  Expression  Exhibition  in  Human  Computer 
Interaction 

William  J.  King  (supervised  by  Suzanne  J.  Weghorst) 

Within  the  context  of  the  evolution  of  the  human-computer  interface,  detection  of 
non-verbal  behavior  of  the  user  may  afford  more  proactive  systems.  In 
particular,  facial  behavior  may  convey  information  about  the  user's  cognitive 
state,  which  can  be  used  to  adapt  system  configuration  and  response.  Facial 
indicators  of  situation  awareness  (or  its  absence)  may  be  useful  metrics,  both  in 
the  laboratory  and  in  deployed  man-machine  systems. 

In  this  study,  facial  expressions  were  recorded  during  subjects'  interactions  with 
a  maze  traversal  game  based  upon  the  towering  inferno  scenario.  Subjects 
were  tasked  with  moving  an  icon  through  a  plan  view  of  a  building  (in  one 
condition),  or  navigating  through  an  ''Adventure"-style  text  scenario  (in  a  second 
condition).  Subjects  were  selected  from  a  large  undergraduate  subject  pool  for 
a  criterion  level  of  computer  experience  using  the  ACM  SIGCHI  computer  user 
abilities  test.  Each  of  the  18  subjects  completed  two  four-minute  trials,  with 
condition  order  counterbalanced  across  subjects. 

Prior  to  each  move,  subjects  were  required  to  request  state  information  about 
the  adjacent  room  ("live"  video  footage,  "live"  audio,  smoke  levels,  temperature, 
or  concentration  of  toxins).  Information  was  redundant  across  information 
modalities  but  all  room  states  were  randomly  re-determined  with  each  icon 
move.  Subject  annoyance  with  system  response  delay  was  also  manipulated 
by  randomly  varying  the  wait  time  for  roorn  state  information  after  each  request. 
In  the  maximal  delay  condition  the  state  information  was  declared  "Unavailable" 
after  a  wait  of  30  seconds.  The  conditions  of  the  game  were  such  that  it  was  not 
possible  to  "win"  within  the  time  frame  allowed.  Subjects  were  shown  an  on¬ 
screen  count-down  clock  throughout  the  task  and  a  final  (non-contingent)  score 
was  flashed  at  the  end  of  the  trial. 

A  telecommunications-style  head-up  display  (reflecting  a  beam-split  image  of  a 
13-inch  color  computer  monitor)  provided  a  means  for  unobtrusive  video  (and 
audio)  recording  of  the  subject's  head  and  upper  body  throughout  the  trials.  All 
keyboard  responses  and  game  parameters  were  continuously  recorded,  along 
with  time  tags,  for  later  micro-interactive  analysis  and  for  correlation  with 
subjects'  non-verbal  behaviors.  Facial  expressions  were  derived  from  the 
acquired  video  sequences  using  the  facial  Action  Unit  coding  methods  of 
Ekman's  Facial  Action  Coding  System  (FACS). 

Initial  results  show  significantly  more  facial  expressions  exhibited  in  the  plan 
view  condition  than  in  the  text-based  condition.  An  unexpected  number  of 
asymmetric  facial  expressions  was  also  observed.  Additional  analyses  are 
undeoAray.  In  particular,  we  will  attempt  to  distinguish  between  perplexity  due  to 
room  state  unpredictability  and  annoyance  due  to  system  delays. 
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2.1.10  Measuring  Presence  in  Terms  of  Subjective  Reference 
Frames 

Jerrold  D.  Prothero  (supervised  by  Donald  Parker  and  Thomas 
Furness) 


Introduction 

•  Does  presence  in  a  virtual  environment  reflect  a  switch  from  using  the  external 
world  to  define  one's  sense  of  position,  orientation,  and  motion  (one's  "rest 
frame")  to  using  cues  provided  by  a  virtual  environment? 

•  This  model  can  be  tested  quantitatively  by  setting  up  an  experiment  in  which 
physical  and  virtual  orientation  cues  conflict,  and  observing  which  cues  subjects 
tend  to  rely  on,  and  whether  the  tendency  to  rely  on  cues  coming  from  the  virtual 
environment  correlates  with  measures  of  presence. 

•  Whether  the  "subjective  rest  frame"  theory  is  shown  to  be  true  or  false,  we  will 
have  learned  something  important  about  the  nature  of  presence. 

•  If  it  is  shown  to  be  true,  the  experimental  paradigm  described  in  the  methods 
section  may  be  useful  for  quantitatively  measuring  the  sense  of  presence  in 
particular  environments. 


Method 

•  Our  apparatus  will  consist  of  a  platform  which  the  subject  will  lie  on  and  which 
can  be  rotated  around  the  Y-axis  running  left-to-right  through  the  subject's  body 
at  the  center  of  mass.  The  subject  will  be  wearing  an  HMD,  and  will  have  a 
pointer  mounted  on  the  platform  near  his  or  her  waist  which  can  be  used  to 
indicate  the  subjective  impression  of  which  direction  is  up. 

•  The  experimental  stimulus  will  consist  of  images  presented  in  the  HMD,  which 
may  be  used  to  provide  a  "visual  polarity"  (indications  about  which  direction  is 
up)  different  from  that  of  external  physical  gravity. 

•  Possible  independent  variables: 

-  Scenes  with  and  without  visual  polarity 

-  Rotating  the  scene  around  the  visual  polarity  or  gravity  axes 

-  Varying  the  field-of-view 

-  Subject’s  tilt  around  the  Y-axis 

•  Possible  dependent  variables: 

-  The  "up"  angle  indicated  by  the  subject 

-  Subjective  measures  of  presence 

-  A  performance  measure  of  presence 

•  Vection: 

-  Latency 

-  Saturation 
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Tilt  Machine 


Expected  Results 


•  If  the  rest  frame  theory  is  correct,  we  expect  that  the  tendency  of  the  subject  to 
identify  "up"  with  the  cues  provided  by  the  virtual  environment  will  correlate  with 
measures  of  presence. 

•  Increasing  the  field-of-view,  which  increases  the  sense  of  presence,  should 
also  increase  the  subject's  tendency  to  identify  "up"  with  the  virtual  cues. 

•  We  suspect  that  rotating  the  scene  will  increase  presence,  and  will  tend  to 
cause  subjects  to  indicate  an  "up"  direction  aligned  with  the  axis  of  rotation. 

Conclusion 

•  The  experiment  will  test  the  simplest  theory  of  what  presence  could  be.  If  the 
theory  is  shown  to  be  false,  we  will  have  a  sound  basis  for  proposing  more 
complex  theories. 

•  If  the  theory  is  shown  to  be  true,  then  we  will  have  a  quantitative  method  for 
measuring  presence,  in  terms  of  the  extent  to  which  we  can  get  subjects  to 
identify  "up"  with  the  visual  polarity  of  the  virtual  scene,  rather  than  with  the 
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physical  gravitational  cues.  This  would  allow  us  to  investigate  quantitatively 
the  importance  of  factors  such  as  field-of-view  and  resolution  to  presence. 
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2.1.11  Navigation  and  Wayfinding  in  VR:  Finding  the  Proper 
Navigational  Tools  and  Cues  to  Build  Navigational 
Awareness 

Glenna  A.  Satalich  (supervised  by  Max  Wells) 

Introduction 

Virtual  Reality  allows  the  participant  to  experience  data  and  environments  in  a 
3-D  and  egocentric  manner.  This  gives  the  participant  multiple  viewpoints  in 
seeing  the  environment  and  interacting  with  it. 

Question  :  Given  a  perfect  VR  system,  would  the  user  have  problems 
navigating  through  a  VR  world  to  reach  a  particular  area?  The  literature  on 
human  behavior  in  navigation  and  wayfinding  in  the  real  world  say  Yes. 

Goal :  To  find  the  proper  navigational  tools  and  cues  to  build  navigational 
awareness  efficiently. 

Task  :  An  informed  search  and  rescue  task  in  a  virtual  towering  inferno 
environment.  The  goal  location  will  be  known  but  the  path  or  paths  will  not  be 
given.  The  time  and  errors  of  navigators  using  differing  tools,  cues  and  display 
mediums  will  be  compared. 

Navigational  Awareness:  The  ability  to  relate  one's  ego-centered  reference 
frame  with  the  world-centered  reference  frame.  The  five  sequential  steps  for 
navigational  awareness  are: 

1.  landmark  recognition:  choosing  stable  and  directional  landmarks 

2.  routes  or  links:  paths  that  connect  landmarks  together 

3.  survey  knowledge:  knowledge  of  Euclidean  distances  between 
landmarks  and  recognition  of  alternate  paths. 

4.  chunking  of  the  environment:  creating  small  areas  from  a  large 
environment. 

5.  nesting  and  transitioning;  nesting  smaller  areas  under  larger  ones  - 
multiple  representations  of  the  same  space. 

Wayfinding:  The  process  used  to  orient  and  navigate.  The  overall  goal  of 
wayfinding  is  to  accurately  relocate  from  one  place  to  another  in  a  large  scale 
space.  The  four  basic  requirements  for  successful  wayfinding  are 

•  knowing  one's  orientation 

•  choosing  an  appropriate  route 

•  being  able  to  monitor  the  route  for  confirmation  of  being  on  the  correct 
route 

•  destination  recognition 

The  Tools  and  Cues:  Real  world  and  virtual  tools  and  cues  will  be 
incorporated  into  the  environment  to  enhance  each  navigational  step.  The  tools 
and  cues  available  are  extensive  and  a  selection  of  likely  candidates  will  be 
made.  The  following  are  a  subset  of  tools  and  cues  that  can  be  used. 
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•  Architectural  cues:  differentiation,  visual  access  and  complexity 

•  Real  world  tools:  sun  or  moon,  compass,  maps,  signs,  verbal 
directions,  video  tour 

•  Virtual  tools:  Visual  or  Audio  breadcrumbs,  agents  (static  and 
mobile),  teleportation,  contour  and  gridlines 

The  Metrics:  The  most  crucial  step  in  this  research  is  understanding  when  a 
navigator  has  reached  a  particular  navigational  awareness  step.  The  following 
metrics  have  been  used  in  previous  navigation  and  wayfinding  studies  in  the 
real  world.  They  will  be  modified  for  usage  in  a  virtual  world.  Additional  metrics 
are  being  researched. 

•  Landmark  observation  test;  The  goal  is  to  have  the  navigator 
choose  stable  and  directional  landmarks  as  opposed  to  transient  landmarks. 

•  Route  recognition;  The  navigator  should  recognize  a  path 
traveled  between  landmarks  and  be  able  to  give  correct  directions. 

•  Landmark  distances:  For  procedural  knowledge  the  distance 
given  should  be  length  of  the  path.  For  survey  knowledge  the  distance  should 
be  Euclidean.  For  chunking  peculiar  biases  will  be  noticed  depending  on 
whether  the  landmark  is  in  the  same  small  area  or  in  a  differing  one. 

•  Path  inference  testing;  For  survey  knowledge  the  navigator 
should  infer  alternative  paths  if  a  common  path  is  blocked. 

•  Directional  pointing:  This  test  can  be  used  for  procedural,  survey, 
and  nest  transitioning. 
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2.1.12  Effective  Information  Presentation  to  Facilitate  Decision 

Making  Under  Uncertainty 

Ryoko  K.  Williamson  (supervised  by  Max  Wells) 

Introduction 

Effective  decision-making  (DM)  processes  in  a  complex  situation  require  the 
maintenance  of  situation  awareness  to  obtain  satisfactory  outcomes.  Decision¬ 
making  could  be  affected  by  uncertainties  due  to  the  nature  of  the  world  which 
is  dynamic  and  complex.  It  has  been  found  that  the  Traditional  DM  theory  which 
is  based  on  mathematical  models  does  not  reflect  the  human  DM  processes 
(naturalistic  DM). 

Research  Questions: 

•  What  types  of  errors  would  we  expect  from  the  use  of  different  types  and 
forms  of  presenting  information  in  a  virtual  environment? 

•  What  is  the  difference  in  making  decisions  between  experts  and  novices 
given  uncertain  information? 

•  In  what  situations,  do  options  of  alternative  actions  help  to  make  decisions? 

Goal:  Identify  types  and  forms  of  information  that  would  facilitate 

naturalistic  decision  making  under  uncertainty  and  develop  an  interface  using 
that  knowledge. 


Decision  Making 

Namralistlfi  OM 

•  Sequential  Processes:  generates  and  evaluates  a  single  option  at  a  time  -> 
Decision  Cycle 


Generate 


T 


Evaluate 

i 

\ 

I _ 

Act 


Naturalistic  Decision  Cycle 

•  Strategy:  “satisficing"(good  enough) 

•  Schema-Driven  reasoning  guided  by  the  knowledge  of  the  decision  maker. 

•  Difference  between  experts  and  novices  is  their  ability  to  assess  situations, 
not  their  reasoning  skills. 

Traditional  DM 

Consequential  Processes:  generate  all  possible  alternatives  and  select  an 
“optimal”  solution. 
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Uncertainty 

Uncertainty  affects  real-world  decision  making  by  interrupting  ongoing  action, 
delaying  intended  action,  and  guiding  the  development  of  new  alternatives. 

Uncertainties  include; 

1)  the  meaning  of  information 

2)  the  value  of  consequences 

3)  act-event  or  event-event  sequences 
4}  the  appropriate  decision  process 

5)  the  ability  to  affect  future  events 

Interesting  Finding;  It  is  suggested  that  some  uncertainty  is  essential  to  good 
decision  making,  because  it  motivates  decision  makers  to  shift  from  automatic  to 
reflective  action,  and  guides  their  search  for  better  solutions. 

Research  Ideas 

Uncertainties  could  be  introduced  by  getting  information  from; 

•  Sensors  (limited  update  rate,  malfunction) 

This  consequentially  introduces  the  time  factor. 

•  Other  participants  (unpredictability  of  their  action) 

Task:  Assess  the  situation  and  navigate  oneself  by  making  appropriate 
decisions  to  accomplish  a  search  and  rescue  task  successfully. 

Independent  Variables: 

•  Forms  of  Information  :  probability  vs.  descriptive 

•  Types  of  Additional  Information  ; 

a.  List  of  Options 

b.  Video  Camera  View  of  Queried  Location 

Pro&osed  Metrics; 

•  Performance  Time 

•  Decision  Errors 

•  Subjective  Situational  &  Decision  Strategy  Assessment  Test  (e.g.  think 
aloud  or  retrospective  self-report) 
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2.2  Creation  of  a  knowledge  base 

The  HITL  Knowledge  Base  is  a  collection  of  electronic  information  available 
online,  as  well  as  papers,  books  and  other  reference  materials  used  to  support 
research  in  advanced  human  interfaces. 

•  The  AFOSR  portion  of  the  Knowledge  Base  is  devoted  specifically  to  materials 
relating  to  Situational  Awareness  and  Presence  in  Virtual  Environments.  It  was 
started  in  1992  and  contains  over  529  journal  articles,  conference  papers  and 
technical  reports.  The  bibliography  was  developed  and  is  maintained  using  the 
ProCite  bibliographic  database.  It  is  also  available  in  various  electronic 
formats,  including  an  HTML  (Hypertext  Mark-up  Language)  document,  access 
to  which  is  available  for  the  AFOSR  Project  group  using  the  Mosaic  interface. 
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How  the  AFOSR  Knowledge  Base  was  constructed 


•  The  first  step  was  to  interview  professionals  in  the  field  to  identify  key 
concepts,  researchers,  conference  proceedings,  and  journals. 

•  Among  the  databases  searched  for  relevant  material  were  INSPEC, 
COMPENDEX,  NTIS  and  PSYCLit. 

•  Relevant  citations  were  identified  and  documents  were  retrieved  and  indexed 
into  ProCite.  The  documents  were  filed  alphabetically  by  first  author. 

•  Many  of  the  articles  in  the  AFOSR  Knowledge  Base  were  also  found  as  the 
result  of  student  research  projects. 

•  In  addition  to  the  formal  identifiers  provided  by  the  commercial  databases, 
such  as  INSPEC,  the  AFOSR  Knowledge  Base  is  also  indexed  informally 
according  to  specific  student  interests.  Each  HITL  student  identified  three 
phrases  relevant  to  their  research  area  of  interest.  The  eight  HITL  students 
reviewed  and  classified  all  of  the  articles  in  terms  of  these  key  words.  The 
results  of  this  classification  exercise  are  also  available  as  part  of  the  AFOSR 
Bibliography.  This  process  was  also  helpful  in  removing  articles  that  were  not 
directly  relevant  to  the  AFOSR  literature  review. 

•  The  bibliography  is  available  online  to  HITL  Staff  and  students  through  the 
information  browser  developed  at  NCSA,  Mosaic. 

The  keywords  for  these  articles  provided  by  databases  such  as  INSPEC  are 
also  available  online  with  abstracts,  so  articles  on  particular  documents  can  be 
identified  quickly. 

•  The  HITL  Knowledge  Base  staff  continues  to  monitor  the  literature  for  new 
resources. 


2.3  Development  of  an  experimental  paradigm 

One  of  the  goals  of  the  research  effort  is  an  investigation  of  situation  awareness. 
In  order  to  do  so,  it  is  necessary  to  create  an  environment  in  which  to 
manipulate  variables,  collect  data  and  test  hypotheses.  The  operational 
environment  in  which  the  Air  Force  is  interested  involves  a  complex  interplay  of 
multiple  elements  in  an  air-to-air  engagement,  or  some  other  command  and 
control  situation.  Simulating  such  an  environment  in  virtual  reality  has  been 
accomplished  by  various  labs,  and  several  of  the  principle  investigators  on  this 
project  have  experience  both  developing  and  working  with  such  scenarios. 
However,  one  drawback  with  such  an  environment  is  the  need  to  use  highly 
skilled  operators  to  collect  meaningful  data.  Also,  the  results  of  such  work 
would  be  context  specific,  whereas  the  aim  of  this  research  project  is  to 
generate  data  and  provide  guidelines  which  will  be  relevant  across  a  number  of 
different  contexts. 
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In  order  to  address  the  needs  of  the  project  and  the  constraints  of  our  subject 
pool,  a  more  generic  scenario  was  developed,  termed  the  "Towering  Inferno.” 


Towering  inferno 


controller 


The  “Towering  Inferno”  consists  of  a  number  of  generic  elements,  illustrated  in 
the  figure.  The  basic  scenario  is  that  of  a  multi-story  building,  only  one  story  of 
which  is  illustrated.  Each  story  consists  of  several  rooms,  some  of  which  may 


be  on  fire.  There  is  an  operator,  whose  task  it  is  to  perform  a  search  and  rescue 
task  (reach  the  objective).  The  operator  may  be  aided  or  hindered  by  agents. 
The  agents  may  be  computer-controlled,  or  manned.  Some  of  the  agents  may 
be  in  direct  communication  with  the  operator  or  may  be  in  contact  via  a 
controller  stationed  outside  the  building.  The  controller  may  also  collect 
information  from  sensors  positioned  throughout  the  building.  The  channels  of 
communication  can  be  selectively  manipulated  (e.g.  degraded  or  delayed)  by 
the  experimenter.  Also,  the  state  of  the  fire  in  each  room  may  be  controlled  by 
the  experimenter.  By  imposing  time  stresses  and  competition,  the  experimenter 
can  create  fairly  demanding  experimental  situations. 

The  “Towering  Inferno"  provides  participants  with  complex  task-demands.  It  is 
capable  of  being  used  by  people  with  varying  skill  levels.  It  can  be  used  by 
multiple  participants,  and  provides  a  range  of  complex  scenarios  with  which  to 
explore  issues  such  as  situation  awareness,  training,  communication,  spatial 
awareness,  command  and  control,  and  decision  making. 


2.4  Hardware 

An  integral  part  of  the  proposal  to  the  AFOSR  was  the  purchase  of  a  SGI  Onyx/2 
RE2  computer  for  the  development  of  complex  virtual  environments.  The 
machine  was  ordered  in  late  May  and  received  in  late  June.  It  has  been 
integrated  into  the  HIT  lab's  computing  infrastructure.  A  complete  list  of  the  HIT 
lab's  facilities  is  shown  in  Appendix  A,  along  with  a  description  of  the  lab.  The 
facilities  being  used  by  the  AFOSR  project  include  those  of  the  Interactive 
Computer  Graphics  and  Human  Factors  Lab,  headed  by  Dr.  Woodrow  Barfield. 
A  description  of  that  lab,  and  it's  facilities,  is  included  in  Appendix  B. 

Another  recent  purchase,  which  will  enhance  the  HIT  lab’s  research 
capabilities,  was  a  light-weight  helmet-mounted  display,  the  VR4  from  Virtual 
Research  Systems  Inc.  The  VR4  is  an  active  Matrix  LCD-based  HMD  with 
742x230  pixels  per  eye  and  a  60-67  degree  diagonal  field  of  view. 

Interpupillary  distance  (IPD)  is  adjustable  between  52mm  and  74mm.  Eye  relief 
is  adjustable  between  10  and  30  mm.  Overlap  is  user  settable  between  85% 
and  100%.  The  device  wili  take  stereoscopic  and  monoscopic  image  sources, 
switched  at  a  control  box.  It  has  input  connectors  for  RS-1 70A  (RGB)  and  S- 
VHS  video  formats.  The  helmet  comes  with  built-in  Sennheiser  HD440 
headphones. 

In  order  to  increase  the  realism  and  inter  activity  of  the  virtual  environments 
which  are  planned  for  this  project,  and  to  expand  on  the  lab's  capabilities,  a 
proposal  was  written  by  the  HIT  lab  to  the  DoD  (U.S.  Army  Research  Office,  Attn: 
AMXRO-IP,  DURIP  proposal  Topic  #11)  for  funding  to  develop  a  virtual  reality 
test  bed.  The  specifications  for  this  device  are  shown  in  Section  2.4.1 . 

The  Neurosciences  Laboratory  at  Johnson  Space  Center  has  agreed  to  provide 
a  90  foot-pound  servo-controlled  rotator  (Contravas)  to  the  HITL  on  long-term 
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loan.  The  rotator  permits  mounting  of  various  restraint  systems  and  locomotion- 
simulation  apparatus.  It  includes  a  chair  that  provides  simple  on-axis  yaw 
rotation.  We  anticipate  that  the  rotator  will  be  used  to  examine  effects  of 
combined  visual  and  inertial  cues  on  various  dependent  variables  including 
self-motion  perception,  measures  of  "presence."  spatial  memory  and 
compensatory  tracking.  A  second  rotator-driven  apparatus  -  The  Pitch  and  Roll 
Device,  PARD  --  may  be  acquired  from  Johnson  Space  Center  at  a  later  date. 


2.4.1  Planned  developments 

HIT  Lab  Testbed 

In  order  to  develop  more  effective  approaches  for  communicating  situation 
awareness  in  virtual  environments,  we  will  be  constructing  a  testbed  for 
exploring  the  sensory,  perceptual,  and  cognitive  factors  which  co''*'’ibute  to  this 
transfer  of  information.  The  HIT  Lab  testbed  will  feature  a  virtual  environment 
display  involving  the  projection  of  computer  generated  stereoscopic  imagery 
onto  one  or  more  screens  which  surround  a  participant  wearing  shutterglasses. 
This  system  will  provide  us  with  an  opportunity  to  investigate  a  wide  range  of 
egocentric  display  concepts.  In  addition,  a  separate  non-projection  monitor  will 
be  included  to  provide  an  exocentric  display. 


HIT  Lab  Testbed;  three  flat  screen  configuration,  with  exocentric  display. 


Goals 

•  Field  of  View:  Vary  instantaneous  FOV  through  180°  and  beyond. 

•  Visual  Acuity:  Support  a  wide  range  of  resolutions,  (ideally)  approaching 
human  visual  acuity  of  about  1  min.  of  arc. 
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•  Latency:  Study  effects  of  eliminating  rotational  latency  while  retaining 
stereoscopic  latency. 

•  Simulated  HMD:  Electronically  blank  portions  of  display,  based  on  head 
orientation. 
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Proposed  Implementations 

Layout  Projection  Graphics 


3  flat  screens 

rear  projection 
-  /arpe  room  size  l30’x2Q') 


/7\ 

180° 


1  flat  screen 

rear  projection 

*  upgradable  m  3  screens 

*  smaller  room  site  (8‘x20') 
■  small  field  of  regard 


o 

60° 


curved  screen 

front  projection 
•  large  room  size  (I6'x20') 


180° 


3  projectors  stereo  3  RE2s 

3  high-end  CRT  projectors:  $20K  x  3  >  [$60K]  upgrade  Onyx  [-$2S0K] 

-  requires  active  eyevrare  (t05‘  IFOV)  3  stereo  outputs,  each  1025x768  96  Hz  (48Hz/eye) 


♦  total  resolution  3075x768:  3.5  rrtin  of  arc  pixels 


6  projectors  stereo 

6  low-end  CRT  projectors:  6  x  -SIOK  =  l'$60K] 

2  projectors/screen 
+  supports  passive  eyevrare  (180”  IFOV) 

*■  brighter 

■  additional  alignment  complexity 

MCO 

already  have  [$0] 

6  outputs,  each  640x480  60Hz  (OOHz^eye) 

-  total  resolution  1920x480:  5.5  min  of  arc  pixels 

■  slower  update  rate  with  i  RE2  than  with  3  RE2s 

3  projectors  mono 

3  low-end  CRT  projectors:  3  x  ~$10K  =  (~$30K] 

*  supports  passive  eyevrare  (180°  IFOV) 

■  not  stereo 

MCO 

already  have  [$0] 

3  outputs,  each  960x680  60Hz 
total  resolution  2880x680:  4  min  of  arc  pixels 

1  projector  stereo 

high-end  CRT  projector:  [$20K] 

-  requires  active  eyevrare  (105°  IFOV) 

1  RE2 

already  have  [SO] 

1  stereo  output  1 025x768  96  Hz  (48Hz/eya) 

*  total  resolution  1025x768:  3.5  min  ol  arc  pixels 

2  projectors  stereo 

2  high-end  CRT  projectors:  2  x  '$20K  =  [~$40K) 

*  supports  passive  eyevrare  (180°  IFOV) 

*  brighter 

■  additional  alignment  complexity 

MCO 

already  have  [SO] 

2  outputs,  each  1 280x1 024  60Hz  (60Hz/eye) 

*  total  resohjtion  1280x1024: 2.8  min  ol  are  pixels 

•  MCO:  cannot  upgrade  to  3  screens  at  1280x1024 

2  projectors  stereo 

2  low-end  CRT  projectors:  2  x  -$10K  » I--$20K) 

*  supports  passive  eyevrare  (180°  IFOV) 

*  brighter 

*  additional  alignment  complexity 

MCO 

alteady  have  [SO] 

2  outputs,  each  640x480  60Hz  (60Hz/eye) 

-  total  resolution  640x480:  S.S  min  of  arc  pixels 

1  projector  stereo  1  RE2 

Hughes-JVC  Light  Amplifier  projector:  already  have  [$0] 

3tOE:  2000  lumens  ($60K]  -or-  1  stereo  output  1025x768  96  Hz  (48Hz/leye) 

320S:  2300  lumens  [$70K]  -or-  ■  total  resolution  1025x768:  10.5  min  of  arc  pixels 

335S:  3500  lumens  [$1 15K]  -  curved  screen:  graphics  computation  slower 

optics  for  curved  screen  [$?] 

♦  MUCH  brighter  than  CRT  projectors 

•  16.7ms  response  lime  probably  insufficient  for  stereo 

-  requires  active  eyeware  (t0S°  IFOV) 

1  projector  mono 

Hughes-JVC  Light  Amplifier  projecror: 
same  as  above:  ($60K,  $70K,  or  $1 15K] 

*  MUCH  brighter  than  CRT prr^aors 

-  not  stereo 

-  requires  active  eyevrare  (105°  IFOV) 

1  RE2 

alteady  have  [SO] 
lou^  1280x102460Hz 
■  total  resolution  1280x1024:  8.5  min  ol  arc  pixels 
-  curved  screen:  graphics  computation  slower 
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Details  of  three  flat  rear-projection  screen  design  with  additional  exocentric  display. 


Equipment 

•  Graphics  computer:  The  HIT  Lab  currently  has  a  Silicon  Graphics 
Deskside  Onyx  RE2,  with  2  processors,  1  Reality  Engine  2  (RE2)  graphics 
board  with  4  raster  managers  (RMs),  and  a  Multi-Channel  Option  Board 
(MCO).  This  system  could  potentially  be  upgraded  to  a  three  RE2  rack¬ 
mounted  system  for  the  planned  three  screen,  three  projector  stereo  testbed 
configuration. 

•  Projectors:  A  variety  of  projectors  are  being  considered  for  inclusion  in  the 
system.  These  vary  in  price  and  performance.  The  final  decision  will  be 
based  on  a  price,  performance  and  upgradability  trade-off. 
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2.5  Development  of  software 

The  development  of  virtual  environments  is  a  complex  task,  requiring  a  unique 
integration  of  hardware  and  software  skills.  Often,  these  skills  are  held  by  more 
than  one  person,  and  so  the  challenge  becomes  one  of  both  technical  and 
managerial  integration.  In  the  past,  building  the  various  components  of  a  virtual 
world  was  the  domain  of  a  few  highly  specialized  individuals.  To  a  certain 
extent  this  is  still  the  case,  however  a  number  of  vendors  have  been  producing 
products  which  address  some  of  the  needs  of  newcomers  to  the  VR  business. 

In  particular,  there  are  toolkits  which  help  in  the  building  of  virtual  worlds. 

In  the  virtual  worlds  which  we  wish  to  use  for  our  experiments  (the  “Towering 
Inferno”),  there  is  a  need  for  complex  scenarios  such  as  live  video,  augmented 
reality,  and  multiple  viewports.  These  cannot  all  be  done  with  commercially 
available  software,  and  so  there  is  a  need  to  develop  our  own.  On  the  other 
hand,  in  order  to  maximally  utilize  the  talents  of  the  students  who  are 
conducting  much  of  the  research,  there  is  a  need  for  well  documented,  well 
supported,  easy-to-use  software.  Also,  in  our  role  as  an  educational 
establishment,  we  need  to  ensure  that  the  software  skills  that  the  students 
acquire  during  their  time  in  the  lab  are  portable  to  the  outside  world.  The  trade¬ 
off  between  ease-of-use  and  functionality  is  one  which  will  continue  to  be  made 
during  the  research  project.  However,  for  the  time  being,  we  have  chosen  to 
use  dVISE,  made  by  Division,  for  our  world  development.  We  are  also 
considering  the  use  of  World  Toolkit,  by  SenseS,  as  another  authoring  package. 

Prior  to  our  decision  to  use  commercial  software,  we  produced  a  proof  of 
concept  demonstration  of  the  “Towering  Inferno”  using  in-house  software.  This 
was  developed  on  DEC  machines  with  Kubota  graphics  board.  The 
demonstration  was  successful,  achieving  a  20  Hz  update  rate,  live  video,  and 
multiple  viewports.  Work  also  started  on  an  experimenter’s  control  station. 
Subsequently,  the  graphics  boards  went  out  of  production  and  were  no  longer 
supported.  Some  of  that  software  is  currently  being  ported  to  SGI  machines. 
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3.0  Cumulative,  chronological  list  of  written  publications 
(including  in  press  and  in  preparation) 


Barfield,  W.,  Bjorneseth,  0.  and  Hendrix.  C.  Spatial  performance  with 
stereoscopic  displays  as  a  function  of  computer  graphics  eyepoint  elevation 
and  geometric  field  of  view,  submitted  to  :  Applied  Ergonomics,  June  1 994. 

Barfield,  W.,  Hendrix,  C.,  Bjorneseth,  O.,  Lotens,  W.,  and  Kazaramac,  K. 
Comparison  of  human  sensory  capabilities  with  technical  specification  of  virtual 
environment  equipment,  submitted  to  :  Presence  Journal,  July  1994. 

Barfield.  W.  and  Hendrix,  C.  Exploratory  studies  on  the  sense  of  presence  within 
stereoscopic  virtual  environments,  submitted  to  :  Presence  Journal,  July  1994 

Barfield,  W  .  and  Weghorst,  S.  The  sense  of  presence  within  virtual 
environments:  A  conceptual  framework,  5th  International  Conference  on 
Human-Computer  Interaction,  Orlando,  Florida,  1 993. 

Barfield,  W.,  and  Danas,  E.  Comments  on  the  use  of  olfactory  displays  for  virtual 
environments,  will  be  submitted  to  :  Presence  Journal,  August  1994 

Hendrix,  C.  and  Barfield.  W.  Spatial  discrimination  in  three-dimensional 
displays  as  a  function  of  eyepoint  elevation.  In  press  :  Human  Factors  Journal. 

Hendrix,  C.  and  Barfield,  W.  Relationship  between  monocular  and  binocular 
depth  cues  for  judgments  of  spatial  information  and  spatial  instrument  design, 
submitted  to:  Displays  Journal,  April  1 994. 

Hendrix,  C.  and  Barfield,  W.  Exploratory  studies  on  the  sense  of  presence  within 
auditory  virtual  environments,  will  be  submitted  to  :  Presence  Journal, 
September  1994 

Hendrix,  C.  and  Barfield,  W.  Exploratory  studies  on  presence  and  spatial 
discrimination  in  virtual  displays  as  a  function  of  geometric  field  of  view, 
headtracking,  and  stereopsis,  will  be  submitted  to  :  Presence  Journal, 
September  1994 

Hoffman,  H.G.,  Hullfish,  K.,  and  Houston,  S.  (in  preparation).  Virtual-reality 
Monitoring.  Paper  to  be  submitted  for  publication. 

Hollander,  Ari  J.  (1994).  Master's  Thesis:  "An  Exploration  of  Virtual  Auditory 
Shape  Perception." 

Hollander,  Ari  J.,  and  Furness,  Thomas  A.  "Perception  of  Virtual  Auditory 
Shapes".  International  Conference  on  Auditory  Display.  Santa  Fe,  NM.  Nov. 
1994.  In  press 
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King,  William  J.  (1993)  "Investigation  of  the  Exhibition  of  Facial  Expressions 
within  Human  Computer  Interaction,"  in  Proceedings  of  the  2nd  IEEE 
International  Workshop  on  Robot  and  Human  Communication.  1993,  182-187. 

King,  William  J.  (1994)  "Defining  Phenomena  for  an  Emotion  State  Model  in  the 
Human  Interface,"  in  Proceedings  of  the  3rd  IEEE  International  Workshop  on 
Robot  and  Human  Communication  ,  1993,  191-195. 

King,  William  J.  (submitted)  "The  Representation  of  Agents,"  in  Proceedings  of 
the  1995  ACM/SIGCHI  Conference  on  Human  Factors  in  Computing  Systems  , 
1995. 


4.0  Professional  personnel  associated  with  the  research  effort 


Staff 


Woodrow  Barfield  (Director,  Interactive  Computer  Graphics  and  Human  Factors 
Lab) 

B.A.  Experimental  and  Engineering  Psychology,  1976,  UCLA. 

Ph.D.  Industrial  Engineering,  1986,  Purdue  University. 

William  Bricken  (on  leave  without  pay  since  March  1994) 

B.A.  Psychology,  1967,  University  of  California  at  Los  Angeles. 

M.A.  (candidate),  1968,  Social  Psychology,  Ohio  State  University,  OH. 
Dip.  Ed.  1972,  Monash  Teachers  College,  Melbourne,  Australia. 

M.Ed  (candidate)  1975,  La  Trobe  University,  Melbourne,  Australia. 

M.S.  Statistics,  1984,  Stanford  University,  CA. 

Ph.D.  Education,  1987,  Stanford  University,  CA,  "Analyzing  errors  in 
elementary  mathematics. 

Toni  C.  Emerson  (Research  Staff,  HIT  Lab) 

B.A.  Spanish;  B.A.  Drama,  1973,  University  of  Washington,  Seattle. 
M.L.S.  (Masters  of  Library  and  Information  Science),  1993,  University  of 
Washington,  Seattle. 

Thomas  A  Furness  III  (Director,  HIT  Lab) 

B.A.  Electrical  Engineering,  1966,  Duke  University. 

Ph.D.  Engineering  and  Applied  Science,  1981,  University  of 
Southampton,  England. 

Donald  E.  Parker  (Adjunct  Professor,  Dept  of  Otolaryngology,  UW) 

B.A.  Psychology/Economics  (1958),  DePauw  University,  IN 
Ph.D.  Experimental  Psychology  (1961),  Princeton  University,  NJ, 
“Vertical  organization  of  the  auditory  cortex  in  the  cat." 
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Hunter  Hoffman  (Research  Staff,  HITL) 

B.S.  Psychology,  1985,  The  University  of  Tulsa,  OK. 

M.S.  and  Ph.D.  1992,  Cognition  and  Perception,  minoring  in 
Physiological  Psychology,  University  of  Washington,  Seattle,  "Reality 
monitoring:  the  process  by  which  people  separate  memories  of  real 
events  (fact)  from  imagined  events  (fantasy)." 

Suzanne  J.  Weghorst  (Director  of  Human  Factors  and  Interface  Development, 
HIT  Lab) 

B.S.  Psychology,  1972,  Seattle  University. 

M.A.  Psychology,  1975,  University  of  California,  Riverside. 

Ph.D.  (candidate),  Psychology,  1977,  University  of  California,  Riverside, 
"A  Sociobiological  Approach  to  Human  Jealousy." 

M.S.,  Computer  Science.  1989,  University  of  Washington,  "Exploring 
Graph  Perception  with  an  Automated  User  Interface  Research  Tool" 

Maxwell  J.  Wells  (Associate  Director,  HIT  Lab) 

B.Sc.  (Honors)  Biology  and  Psychology,  1978,  University  of  Stirling, 
Scotland. 

Ph.D.  Engineering  and  Applied  Science,  1983,  University  of 
Southampton,  England,  "Vibration'induced  eye  movements  and  reading 
performance  with  the  helmet-mounted  display." 

CPE  (Certified  Professional  Ergonomist)  1993. 


Students 

Ove  L.  Bjorneseth  (Research  Assistant,  Laboratory  for  Interactive  Computer 
Graphics  and  Human  Factors) 

B.S.  Electrical  Engineering,  1985,  Gjovik  College  of  Technology, 
Norway. 

B.S.  Industrial  Engineering,  1992,  University  of  Washington,  Seattle. 
M.S.E.  Human  Factors,  In  Progress,  University  of  Washington,  Seattle. 

Gina  B.  Crvarich  (Research  Assistant,  HIT  Lab) 

B.Sc.  Information  and  Computer  Science,  1985,  University  of  California, 
Irvine. 

M.S.E.  Human  Factors,  In  Progress,  University  of  Washington,  Seattle. 
Mark  H.  Draper  (Captain,  USAF) 

B.S.E.  (Magna  Cum  Laude)  Human  Factors  Engineering,  1989,  Wright 
State  University,  Dayton,  Ohio. 

M.S.E.  Human  Factors,  In  Progress,  University  of  Washington,  Seattle. 

Robert  G.  Futamura  (Research  Assistant,  HIT  Lab) 

B.A.  Physics  and  Psychology,  1992,  Miami  University,  Oxford,  Ohio. 
M.S.E.  Human  Factors,  In  Progress,  University  of  Washington,  Seattle. 
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Claudia  Hendrix.  (Research  Associate,  Laboratory  for  Interactive 
Computer  Graphics  and  Human  Factors) 

B.S.  Industrial  Engineering  and  Management,  1986,  North  Dakota  State 
University,  Fargo. 

M.S.I.E,  1994,  University  of  Washington,  Seattle.  "Evaluation  of  Presence 
in  Virtual  Environments  as  a  Function  of  Visual  and  Auditory  Cues”. 

Ph.D.  Industrial  Engineering  and  Human  Factors,  In  Progress,  University 
of  Washington,  Seattle. 

Ari  Hollander,  (Research  Assistant,  graduated,  HIT  Lab,  currently  HIT  Lab  staff) 

A. B.  Astrophysics,  1991,  University  of  California. 

M.S.I.E.  (Virtual  Environments  and  Advanced  Interface  Technology), 
1994,  University  of  Washington,  Seattle,  "An  Exploration  of  Virtual 
Auditory  Shape  Perception." 

Keith  C.  Hullfish  (Research  Assistant,  HIT  Lab) 

B. A.  Mathematics,  Physics  concentration,  1987,  Franklin  and  Marshall 
College,  Lancaster,  Pennsylvania. 

M.S.E.  Human  Factors,  In  Progress,  University  of  Washington,  Seattle. 

William  J.  King  (Research  Associate,  Psychology) 

B.A.  (Honors),  1993,  Experimental  Psychology  and  Computer  Science, 
Southwestern  University  (Georgetown,  Texas) 

Ph.D.  Cognitive  Psychology,  In  Progress,  University  of  Washington, 
Seattle. 

Jerrold  D.  Frothero  (Research  Associate,  HIT  Lab) 

B.S.  (Honors)  Physics  and  Computer  Science,  1986,  University  of 
Washington,  Seattle. 

M.S.E.  Interdisciplinary  Engineering,  1993,  University  of  Washington, 
Seattle,  "The  Treatment  of  Akinesia  Using  Virtual  Images." 

Ph.D.  M.E.,  In  Progress,  University  of  Washington,  Seattle. 

Geoffrey  M.  Silverton  (Research  Assistant,  HIT  Lab) 

Sc.B.  (Honors)  Electrical  Engineering,  1991,  Brown  University, 
Providence,  Rl. 

M.S.E. E.,  1993,  University  of  Michigan,  Ann  Arbor. 

Ph.D.  E.E.,  In  Progress,  University  of  Washington,  Seattle. 

Glenna  A.  Satalich  (Research  Assistant,  HIT  Lab) 

B.A.  Experimental  Psychology,  1982,  University  of  California,  San  Diego. 
M.S.E.  Human  Factors,  In  Progress,  University  of  Washington,  Seattle. 

Ryoko  K.  Williamson  (Research  Assistant,  HIT  Lab) 

B.S.  (Honors)  Cognitive  Science,  1993,  University  of  Washington, 

Seattle. 

M.S.E.  Human  Factors,  In  Progress,  University  of  Washington,  Seattle. 
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5.0  Interactions 


5.1  Papers  presented  at  meetings 

Hendrix,  C.  and  Barfield,  W.  Perceptual  biases  in  spatial  judgments  as  a 
function  of  eyepoint  elevation  angle  and  geometric  field  of  view.  The  5  th 
European  Symposium  on  Space  Environmental  Control  Systems, 
Friedrichshafer,  Germany,  June  20-23,  1994. 

Hoffman,  H.,  Hullfish,  K.,  and  Houston,  S.  (July.  1994).  Confusions  between 
memories  of  real,  virtual  and  imagined  events.  Paper  presented  at  AFOSR 
workshop,  HITL,  Seattle  WA. 


5.2  AFOSR  workshop 

On  July  14-15,  1994,  the  Human  Interface  Technology  Laboratory  (HIT  Lab)  at 
the  University  of  Washington  hosted  a  workshop  to  share  research  ideas  and 
agendas,  and  to  promote  cooperation  between  virtual  reality  research  programs 
sponsored  by  the  Air  Force.  Representatives  from  various  Air  Force 
organizations  and  researchers  from  the  University  of  Washington  attended.  The 
following  people  attended  the  workshop: 


Name 

Organization 

Tom  Furness 

UW  /  HITL 

John  Tangney 

SAF/AQT 

Bob  Eggleston 

AL/CFHP 

Bill  Reedy 

SM-ALC  /  FMD  (2) 

Richard  Thurman 

AL / HRA 

Dan  Repperger 

AL/CFBS 

Paul  V.  Whalen 

AL / CFBA 

Art  Kerr 

HITL 

Dick  Slavinski 

RL/C3BA 

Long  Yu 

NAWCWPNS  / 
P237 

Suzanne  Weghorst 

UW/HITL 

Jim  Fleming 

AL/HRTI 

Bill  Berry 

AFOSR  /  NL 

Maxwell  Wells 

UW/HITL 

Wayne  L.  Waag 

AL/HRA 

Glenna  A.  Satalich 

UW  /  HITL 

Ryoko  Williamson 

UW  /  HITL 

Gina  Crvarich 

UW/HITL 

Robert  G.  Futamura 

UW/HITL 

Jerry  Prothero 

UW/HITL 

Geoffrey  Silverton 

UW  /  HITL 

E-mail 

tfurness@u.washington.edu 

(703)  746-8900 

teggleston@falcon. 

reedybi@smdi501.mcclellan.af.mil 

thurman@hrlban1  .aircrew.asu.edu 

d.repperger@ieee.org 

p.whalen@ieee.org 

adkerr@hitl.washington.edu 

slavinskir@lonex.rl.af.mil 

yul@mugu.navy.mil 

weghorst@u.washington.edu 

fleming@alr+d.af.mil 

berry@afosr.af.mil 

mwells@hitl.washington.edu 

waag@hrlban1  .aircrew.asu.edu 

geesat@hitl.washington.edu 

ryokow@hitl.washington.edu 

crvarich@hitl.washington.edu 

futamura@u.washington.edu 

prothero@hitl.washington.edu 

geoffs@hitl.washington.edu 
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Claudia  Hendrix 


UW  /  Human 
Factors  Lab 
AFIT/ENG 


Capt.  Keith  A. 
Shomper 
Ove  Bjorneseth 

Keith  C.  Hullfish 
Woody  Barfield 

Stacey  J.  Houston 
Capt.  Mark  Draper 
Hunter  Hoffman 


UW  /  Human 
Factors  Lab 
UW/HITL 
UW/ Human 
Factors  Lab 
UW  /  HITL 
UW 

UW  /  HITL 


hendrix@u. washington.edu 

kshomper@afit.af.mil 

ovebj@u. washington.edu 

eagle@hitl.washington.edu 
barfield@u.  washington.edu 

stacey-houston@uwtc.washington.edu 
drapermh@u. washington.edu 
hunter@hitl.washington.edu 


The  workshop  consisted  of  presentations  by  each  of  the  Air  Force 
representatives  as  well  as  a  description  of  activities  at  the  University  of 
Washington  being  supported  by  the  AFOSR  grant.  Items  discussed  appear  on 
the  following  agenda. 


USAF/U  of  WASHINGTON 
WORKSHOP  ON 
VIRTUAL  REALITY 


Thursday,  ll  Jiijy 


0800 

CONTINENTAL  BREAKFAST 

(1st  Floor  Conference  Room,  Fluke  Hall) 

0830 

Welcome  and  Opening  Remarks 

Waag  AL/HRA 

0840 

Overview  of  the  HIT  Lab 

Furness 

UW 

0900 

Virtual  Reality  as  a  Performance  Interface  and  a 

Regian 

Training  Tool:  Empirical  Results 

AL/HRT 

0920 

Virtual  Environments  and  Distributed  Interactive 

Thurman 

Simulation:  Aircrew  Training  Research  at 
Armstrong  Laboratory 

AI/HRA 

0940 

Overview  of  VR  at  AFIT 

Shomper 

AFIT 

1000 

BREAK 

1020 

Assessment  of  Human  Performance  with  VR 

Eggleston 

Systems 

AUCFHB 

1040 

Air  Force  Research  in  Human  Sensor  Feedback 

Whalen 

for  Telepresence 

AL/CFBA 

1100 

Overview  of  VR  at  Sacramento  Air  Logistics 

Reedy 

Center 

SM 

ALC/FMDD 

1120 

Overview  of  VR  at  CFBS 

Repperger 

AUCFBS 

1140 

Advanced  Display  and  Intelligent  Interfaces 

Slavinski 

RL/C3AB 

1200 

LUNCH  (Faculty  Club) 

1300 

AFOSR  research  agenda 

Wells 

UW 

1320 

Demonstrations  in  the  HIT  Lab 

1440 

Education  and  training  activities  at  HITL 

Winn 

UW 

1500 

BREAK 

1520 

Force  displays 

Hannaford 

UW 

1540 

Development  of  subjective  measures  of 

Hendrix 

presence  /  Research  in  the  Lab  for  Interactive 

Barfield 

Computer  Graphics  and  Human  Factors 

UW 

1600 

Using  facial  expressions  as  a  metric  of  cognitive 

Weghorst 

function 

UW 

1620 

Virtual  Reality  Monitoring 

Hoffman 

Hullfish 

Houston 

UW 

1640 

Visit  to  the  Laboratory  for  Interactive  Computer 

Barfield 

Graphics  and  Human  Factors 

Friday  15  July 

UW 

0800 

CONTINENTAL  BREAKFAST 

(1st  Floor  Conference  Room,  Fluke  Hall) 

0830 

Review  of  Future  Directions 

All 

0930 

Identify  Possible  Areas  of  Collaboration 

All 

1000 

BREAK 

1020 

Final  Discussions 

All 

1130 

Adjourn 

Graduate  students  working  on  the  grant  also  held  a  poster  session  which 
described  their  thesis  work  and  shared  ideas  with  the  Air  Force  representatives. 
Attendees  also  had  the  opportunity  to  view  demonstrations  of  other  lab  projects. 

POSTER  PRESENTATIONS 

Gina  Cvarich  Essential  characteristics  of  a  graphical  display 

needed  to  communicate  spatial  awareness 

Mark  Draper  The  influence  of  a  virtual  body  on  spatial  awareness 

Toni  Emerson  et  al  Transforming  information  into  knowledge 

Bob  Futamura  Effects  of  spatialized  sound  on  presence 
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Virtual  Reality  monitoring 


Keith  Hullfish 
Hunter  Hoffman 
Stacey  Houston 

Jerry  Prothero 

Glenna  Satalich 

Geoff  Silverton 

Ryoko  Williamson 


Is  presence  a  switch  in  rest  frame? 

Navigating  and  wayfinding  in  VR 

HITLab  Testbed  specifications 

Exploring  the  effect  of  information  representation  for 
decision  making  in  uncertain  situations. 

miiat?  DEMONSTRATIONS 


Greenspace 
Towering  inferno 
Sun  demo 
VRD 

Polyshop 
Virtual  Vision 
Virtual  Lab 

Under  sea  visualization 
Ship  fly  through 
Vestibular  simulator 
Knowledge  base 


Paul  Danset 
Geoff  Silverton 
Wendy  Sommerset 
Rich  Johnston 
Peter  Oppenheimer 
Mark  BiHinghurst 
Dace  Campbell 
Mark  Cygnus 
Mark  Cygnus 
Jerry  Prothero 
Toni  Emerson 


The  workshop  accomplished  several  things.  First,  attendees  acquired  a  survey 
of  the  current  research  projects  at  each  organization.  This  gave  researchers  the 
opportunity  to  establish  communications  with  others  who  had  related  interests. 
Moreover,  formative  discussions  led  to  the  identification  of  needs  for  future 
research  and  collaboration.  It  was  generally  agreed  that  there  was  a  need  for 
benchmark  tasks  in  experiments  whose  results  could  transfer  to  real  world  Air 
Force  scenarios.  Such  tasks  should  reflect  new  models  and  metrics  of  human 
performance  in  VR  that  also  need  to  be  developed.  Lastly,  the  workshop 
provided  a  forum  in  which  to  identify  possible  collaborations.  Participants  had 
the  opportunity  to  share  their  hardware  and  software  needs  which  could  be 
satisfied  by  loans  from  other  organizations. 


6.0  Additional  statements 

A  one  year  hiatus  in  the  HIT  Lab,  during  which  there  were  no  graduate  students, 
meant  that  we  were  unable  to  commence  research  as  soon  as  funding  was 
received.  We  took  on  new  graduate  students  at  the  start  of  the  contract.  They 
spent  theT  first  year  being  trained.  As  is  evident  from  the  number  of  research 
efforts  currently  undenway,  they  are  now  becoming  maximally  productive.  We 
do  not  expect  this  to  have  a  significant  effect  on  our  fulfillment  of  the  contract. 
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Other  significant  events  include  changes  in  personnel.  William  Bricken  stopped 
working  on  the  project  in  February  1994.  Maxwell  Wells  joined  the  HIT  Lab  as 
the  Associate  Director  in  April  1994  and  was  delegated  responsibility  for 
management  of  the  AFOSR  project.  Dr.  Wells  previously  worked  at  Wright 
Patterson  Air  Force  Base  where  he  served  as  a  senior  scientist  on  projects 
related  to  the  use  of  VR  technology  in  high-performance  aircraft. 
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APPENDIX  A 

Description  of  the  Human  interface  Technology  Laboratory 

(HIT  Lab) 


Lab  Description 


Brief  History 

he  HIT  Lab  was  established  in  October  1989  by  the  Washington  Technology 
Center  to  transform  virtual  world  concepts  and  research  into  practical, 
economically  viable  technology  products.  The  Washington  Technology  Center 
allocated  $250K  of  seed  funding  to  the  HIT  Lab  for  the  89-91  biennium.  The  lab 
has  since  grown  and  currently  has  an  operating  budget  of  $7.3  million  for  the 
93-95  biennium. 

Mission 

The  HIT  Lab's  mission  is  to  conduct  research  and  development  in  human- 
machine  interface  technologies  that  will  empower  humans  by  creating  better 
ways  of  interacting  with  advanced  computers  and  other  systems. 

These  interface  technologies  will: 

•  accelerate  learning 

«  enhance  creative  abilities 

•  extend  communication 

•  assist  rapid  information  assimilation 

•  recapture  “lost”  world  citizens 


Personnel  -  Summary 

The  lab  currently  employs  61  people  whose  positions  fall  into  the  following 
categories. 

21  HIT  Lab  Staff 

7  Faculty  Associates 

22  Graduate  student  Asst./Assoc. 

3  Student  Aides 

6  Summer  Interns 
11  Graduates 

4  Visiting  Scholars 
2  Industrial  Fellows 
2  Consultants 


61  TOTAL 
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The  Virtual  Worlds  Consortium 

Organized  in  September  of  1 990,  the  consortium  serves  as  a  nexus  for 
coordinating  the  birth  of  a  new  global  industry.  The  HIT  Lab  serves  as  a  bridge 
between  academia  and  industry,  facilitating  the  flow  of  ideas,  students, 
intellectual  property,  and  patents  between  academia  and  industry.  Consortium 
members  provide  resources  to  the  HIT  Lab  to  develop  infrastructure  and  crucial 
technologies,  and  also  provide  a  focus  and  driving  problems  for  the  lab’s 
activities. 


Current  Projects 

Virtual  Retinal  Display.  The  Virtual  Retinal  Display  (VRO)  will  render 
existing  head-mounted  virtual  reality  systems  and  state-of-the-art  HD  TV 
obsolete.  Rather  than  wearing  a  bulky  helmet  display  or  sitting  in  front  of  a  flat, 
two-dimensional  screen,  the  user  will  put  on  a  pair  of  conventional  eyeglasses 
and  will  view  images  projected  directly  onto  the  retina  of  each  eye.  The  VRD 
can  be  used  to  produce  a  circumambient  environment-a  portable  IMAX  theater 
for  the  eyes. 

Recent  progress  on  the  VRD  project  includes  putting  the  project  infrastructure  in 
place,  successfully  demonstrating  the  first  phase  prototype,  and  the 
development  and  disclosure  of  the  mechanical  resonance  scanner. 

Micro  Vision  Inc.  has  recently  financed  the  commercial  development  of  VRD  by 
the  multidisciplinary  HIT  Lab  VRD  team.  The  company  expects  commercial 
products  from  VRD  technology  to  start  appearing  within  two  years,  and  full 
development  of  the  VRD  technology  to  be  completed  by  1998.  Imaging 
applications  of  the  VRD  will  benefit  science,  medicine,  education, 
telecommunications  and  visual  entertainment. 

Dynamics  Toolkit.  A  Dynamics  toolkit  is  being  created  to  allow  users 
and  world  builders  to  incorporate  physical  dynamics  into  VR  applications. 
Dynamical  simulation  is  important  in  many  application  areas  where  physical 
realism  is  desired  and/or  necessary.  This  software  will  allow  untrained  users  to 
generate  physical  systems  capable  of  performing  complex  motions.  A  menu  of 
various  joints  and  bodies  will  be  presented  to  the  user,  from  which  the  user  can 
construct  his/her  physical  system. 

Spatial  Sound  Server.  The  Spatial  Sound  Server  takes  the  next 
logical  step  in  virtual  audio  display  implementations.  Current  systems  use 
dedicated  hardware  at  each  participant's  VR  workstation.  The  Spatial  Sound 
Server  will  produce  the  same  results  without  the  requirement  of  multiple 
dedicated  audio  hardware. 

Education.  The  HIT  Lab  and  the  U.W.  College  of  Education  joined 
resources  to  establish  the  Learning  Center  as  a  nexus  for  VR  related  research. 
This  exploration  process  ranges  from  introducing  VR  into  school  curriculum  to 
assessing  innovative  strategies  that  promote  learning.  The  Learning  Center  is 
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a  place  where  teachers,  students,  and  research  faculty  come  together  to  build 
virtual  worlds  that  are  relevant  to  school  curriculum  and  to  test  their 
effectiveness  in  the  classrooms.  Beginning  this  Fall,  the  Center  will  act  as  a 
distribution  hub  from  which  the  hardware  and  software  for  world  development 
will  be  sent  out  to  schools  around  the  country  (the  VRRV  Program). 

The  Virtual  Reality  Roving  Vehicles  (VRRV)  Program.  The 
VRRV  is  a  mobile  experiential  learning  laboratory  which  uses  virtual  reality  to 
immerse  students  within  interactive  3-D  VR  learning  environments.  The  primary 
objective  of  the  VRRV  program  is  to  create  the  ultimate  experiential  learning 
environment.  This  environment  will  be  widely  accessible  to  teachers  and 
students  and  will  be  a  place  where  children  learn  by  exploring  and  sharing 
virtual  "worlds".  Students  will  construct  such  worlds  as  part  of  their  K-12  school 
curricula. 

In  the  tradition  of  the  "bookmobile",  the  VRRV  will  transport  the  virtual  reality 
technology  and  worlds  to  the  communities  and  school  settings  across  a  wide 
region,  ensuring  that  a  more  diverse  population  of  students  and  teachers  are 
involved.  Over  the  next  3  years,  four  VRRVs  will  be  deployed  in  four  states. 
Gradually,  the  schools  will  be  linked  with  wide  bandwidth  cables  and  will  be 
able  to  communicate  with  one  another. 

The  knowledge  gained  from  the  VRRV  program  will  help  provide  direction  to 
HIT  Lab’s  Learning  Center  for  future  school-based  VR  projects,  leading  toward 
the  eventual  use  of  VR  in  the  nation's  schools  to  improve  education. 

TreeTown.  Complex  virtual  environments  require  powerful  construction 
techniques.  The  goal  of  the  Treetown  project  is  to  develop  algorithm-based 
construction  tools  for  building  virtual  worlds.  Treetown  is  the  HIT  Lab's 
contribution  to  the  Polyshop  project  being  developed  at  the  University  of  South 
Florida.  Treetown  provides  rapid  modeling  capabilities  that  bridge  the  gap 
between  photographic  images  and  solid  modeling  with  polygons. 

Parkinson's  &  Virtual  Images.  HIT  Lab  researchers  have  helped  to 
develop  a  radical  new  approach  to  the  treatment  of  certain  common  movement 
disorders.  Many  people  with  Parkinson's  Disease  experience  progressively 
greater  difficulty  initiating  and  sustaining  walking,  a  condition  known  as 
akinesia.  Using  Virtual  Vision  Sport  eyewear  co-invented  by  Tom  Furness, 
near-normal  walking  can  be  elicited  by  presenting  virtual  objects  and  abstract 
visual  cues  moving  through  the  patient's  field  of  view  at  speeds  that  emulate 
normal  walking. 

Telemedicine  Interfaces.  Developments  in  medical  imaging  and 
telecommunications  over  the  past  few  years  are  making  real-time  delivery  of 
health  care  over  long  distances  a  viable  option  (e.g.,  long  distance  transmission 
of  digitized  radiology  images  or  video  conferencing  among  doctors).  Spurred 
on  by  the  new  ARPA  program  in  remote  battlefield  trauma  care.  HIT  Lab 
researchers  have  been  exploring  the  possibilities  of  merging  virtual  interface 
technology  with  other  enabling  telemedicine  technologies.  Members  of  the  HIT 
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Lab  have  been  instrumental  in  assembling  an  interdisciplinary  team  of  medical 
researchers  from  the  U.W.  Medical  Center,  the  U.W.  College  of  Engineering  as 
well  as  medical  and  technical  collaborators  outside  the  University.  Several 
research  proposals  have  been  submitted  for  funding. 

GreenSpace.  The  goals  of  this  project  are  to  develop  and  demonstrate  an 
immersive  communication  medium  where  distant  participants  feel  a  sense  of 
presence  in  a  shared  virtual  environment,  a  "virtual  common”.  Ultimately,  the 
project  aims  to  promote  collaboration-at-a-distance  among  100  or  more 
participants  over  broadband  networks  such  as  SONET/ATM,  immersed  in  an 
environment  rich  in  visual,  aural,  and  tactile  cues 

This  fall  the  lab  will  demonstrate  a  preliminary  testbed  in  which  two  to  four 
people,  in  Tokyo  and  Seattle,  will  perform  a  a  cooperative  task  in  a  real-time  3- 
D  visual  space. 

VR-Architecture.  The  CEDeS  lab  (Cascadia  Community  and 
Environment  Design  and  Simulation  Laboratory)  is  a  design  and  simulation 
laboratory  being  put  together  by  the  HIT  Lab  and  the  U.W.  Architecture 
Department.  It  will  integrate  traditional  3-D  design  skills  with  virtual  reality 
technology  to  allow  immersive  simulations  of  planned  buildings  and  proposed 
urban  developments  such  as  the  Seattle  Commons  project.  (A  plan  to  renovate 
a  large  piece  of  waterfront  property).  Graduate  students  from  the  HIT  Lab  and 
the  U.W.  Architecture  Department  have  prepared  a  virtual  representation  of 
what  downtown  Seattle  will  look  like  if  the  Seattle  Commons  proposal  is 
approved.  This  CEDeS  lab  proof  of  concept  demonstration  will  be  presented  to 
the  public  this  Autumn.  The  Virtual  Commons  is  indicative  of  the  kind  of  design 
and  representation  that  will  become  commonplace  with  the  integration  of  VR 
and  architectural  design.  Eventually,  broadband  communication  links  will  allow 
VR  demonstrations  in  the  Architecture  building  using  reality  machines  based  in 
the  HIT  Lab. 

VR  in  Manufacturing.  In  collaboration  with  the  HIT  Lab,  members  of 
the  Industrial  Engineering  department  at  the  University  of  Washington,  applied 
for  and  received  seed  funding  to  investigate  the  use  of  VR  in  manufacturing. 


HIT  Lab  Facilities 
VR  system 

1  Division  Provision  200 


VR  transducers 

3  6D  Joystick 

1  BioMuse  :  bioeiectronic  signal  processor  to  use  nervous  system  signals 
for  the  control  of  digital  interface  devices 
1  Konami  Laserscope  :  voice  command  optical  targeting  headset 
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4  Logitech  6D  mouse  and  tracking  system 
1  Nintendo  PowerGlove 
3  Polhemus  3-Space  Isotrak 
3  Polhemus  3-Space  Fastrak  with  4  sensors 
1  Spaceball  6D 

1  VPL  DataGlove 

VR  output  devices 

2  Crystal  River  Engineering  Convolvotron  3D  sound  systems 

1  Gehring  Focal  Point  3D  sound  card 

2  StereoGraphics  CrystalEyes  3D  glasses 

1  Virtual  Research  VR4  Head-Mounted  Display 
1  Virtual  Research  Flight  Helmet  Head-Mounted  Display 
1  VPL  Eyephones  Head-Mounted  Display 

Network  servers 

1  DECstation  3000  alpha  ;  primary  file  server 
1  DECstation  3000  alpha  :  network  and  e-mail  gateway 
1  DECstation  5000/240  :  X  terminal  server 
1  Sun  SparcStation  2  :  WWW  server 
Ethernet  network  devices 

Lab  computer  systems 

Computers 

t  Silicon  Graphics  100  MHz  R4400  Onyx/2  Reality  Engine2 
1  Silicon  Graphics  ISOMHz  R4400  lndigo2 
1  Silicon  Graphics  4D  320  VGX 

1  Silicon  Graphics  R4600  Indy 

2  Sun  SparcStation  10 
1  Sun  SparcStation  2 
6  Sun  SparcClassic 

5  Sun  4/110 

6  DEC  3000  (4  with  Kubota  video  board,  1  with  6300  board 
samples  speech/video) 

5  DEC  VT1300 

1  DECpc  560  ST 

2  DECpc  433 

1  DECpc  333  portable 
1  DECpc  320  SX  notebook 
5  PC  486 

1  NeXT  workstation 

2  Apple  Macintosh  Quadra  840  AV 
1  Apple  Macintosh  Quadra  660  AV 
1  Apple  Macintosh  Quadra  650 

5  Apple  Macintosh  Quadra  610 
1  Apple  Macintosh  Centris  650 
4  Apple  Macintosh  Centris  610 

3  Apple  Macintosh  llfx 
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1  Apple  Macintosh  Duo  Powerbook 
1  Apple  Macintosh  Powerbook  540C 
1  Apple  Macintosh  Powerbook  520C 
7  Apple  Macintosh  Plus 
1  Commodore  Amiga  3000 

Printers 

1  Apple  LaserWriter  II  printer 
1  Apple  printer 
1  DEC  Laser  3250  printer 
1  DEC  Laser  3200  printer 
1  DEC  PrintServer  20 
1  Hewlett  Packard  DeskJet  650C  plotter 
1  Hewlett  Packard  DeskJet  520 
1  Hewlett  Packard  Thinkjet 
1  NeXT  laser  printer 
1  Tektronix  color  printer 

External  hard  disks 

1  Apple  80SC  expansion 

4  Digital  storage  expansions 
6  Ehman  external  hard  disks 

2  Sun  disks  storage 

External  drives 

2  44MB  Syquest  drives 
1  88MB  Syquest  drive 

1  Ocean  \/256  external  3.5"  drive 

3  CD-ROM  drives 

Scanner 

1  Digital  MD  30C  scanner 
1  ScanPlus  Color  3000  scanner 

Software 

Development 

Borland  C++  (PC) 

C  compiler  (DEC) 

HyperCard  (Mac) 

Graphics 

Modeling,  rendering 

Sense8  WorldToolKit  (SGI)  :  C  libraries  for  building  virtual 

environments  with  texture  maps 

dVISE  (PC,  SGI) :  set  of  high-level  customizable  inclusive 

tools  for  building  virtual  environments 

Alias  (SGI)  :  3D  modeling  and  animation  package 

3D  studio  (PC)  :  CAD  3D  modeling,  rendering  and 

animation 
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AutoCAD  for  Windows  (PC)  :  CAD  drawing  and  3D 
modeling 

Design  Workshop  (Mac)  :  3D  modeling  for  conceptual 

design  and  design  development 

Sketch  :  3D  illustration  and  design 

Swivel  Pro  (Mac)  :  3D  modeling  and  rendering  for  building 

virtual  environments 

Body  Electric  (Mac)  :  Data  flow  language 


Others 

Animator  Pro  (PC)  :  2D  image  and  animation  editing 
Microsoft  Graph  (Mac)  ;  graphs  building  tool 
Mac  Paint  (Mac)  :  bitmap  drawing 
Mac  Draw  (Mac)  ;  object  oriented  drawing 
TypeStyler  (Mac) 

VEOS;  In-house  developed  Virtual  Environment  Operating 
System 


Simulation 

Performer  (SGI)  ;  High  performance  simulation  environment 

Inventor  (SGI)  :  3D  object  oriented  toolkit 

dVS  (PC,  SGI)  :  runtime  software  for  virtual  environments  built  with 

dVISE 

Virtual  Environment  Navigator  (PC)  :  low  and  garage  VR  package 
Rend  386  (PC) :  garage  VR  package 
VR  386  (PC) ;  garage  VR  package 

Scientific  visualization 

StatView  (Mac) ;  statistical  analysis 
Mathematics  (Mac) :  mathematics  package 

AudioA/ideo 

FusionRecorder  (Mac) :  video  frame  grab  to  Quicktime 
Audio  Pointer  (Mac)  :  demonstration 
Midi  3D  (Mac) 

PatchBay  (Mac) :  MIDI  environment 
AudioMedia  (Mac)  :  signal  processing 
BioMuse  software  (PC) 

MAX  (Mac) :  object  oriented  MIDI  development 

AudioA/ideo  equipment 
Video 

Memorex  3.5"  portable  B&W  TV 
Mitsubishi  27"  color  TV 
Panasonic  HiFi  VHS  player 
Panasonic  VHS  player 
Panasonic  video  camera 
Sony  SSM-121  B&W  monitor 
Sony  video  8mm  video  player 
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Sony  video  8mm  camera 

Virtual  Vision  Sport  Personal  projection  TV  and  camcorder 
Phillips  CD-I  System 


Audio 

DEC  Audio  system 
DECTalk 

Stereo  computer  speaker  system 

MacFace  Sonus  midi  interface  used  with  Mac  llfx 

Realistic  stereo  mixing  console 

Tascam  MM-1  keyboard  mixer 

Sony  AM/FM  receiver 

Yamaha  stereo  receiver 

Headphones  with  microphone  (3) 

Headphones  (2) 

Spatial  orientation  manipulation  equipment 
HITL  Tilt  Device 

Contravas  ;  servo-controlled  whole-body  rotator 
Electronic  test  equipment 
Optical  equipment 
Fabrication  equipment 
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APPENDIX  B 


Description  of  the  Laboratory  for 
Interactive  Computer  Graphics  and  Human  Factors 
(Human  Factors  Lab) 


Lab  Description 

The  main  goals  of  this  laboratory  are  twofold;  (1)  to  develop  innovative  techniques  for 
presenting  visual,  auditory,  and  tactile  information  to  users  of  complex  systems  and, 

(2)  to  perform  empirical  studies  to  test  alternative  methods  for  designing  graphical  user 
interfaces.  In  addition  to  the  basic  research  orientation,  we  also  design  and  build 
prototypes  of  visual  and  auditory  interfaces  for  virtual  environment  displays. 

Presence  Within  Virtual  Environments 

The  degree  of  presence  that  someone  feels  in  a  virtual  environment  is  of  major 
interest  to  researchers  and  designers  of  virtual  worlds.  The  basic  systems  we  use  for 
creating  stereoscopic  virtual  environments  and  to  study  presence  includes  Silicon 
Graphics  (SGI)  workstations  to  model  and  render  images,  a  6  ft  by  8  ft  rear  projection 
system  to  view  the  images,  and  a  medium  resolution  head-mounted  display  for  full 
immersion.  With  the  large  rear-lit  screen  viewing  system,  we  use  Stereographies 
shutter  glasses  to  view  the  stereo  images  rather  than  a  head-mounted  display  (HMD). 
Using  this  system,  we  achieve  a  high  spatial  resolution  (1024  x  1280)  but  at  the 
expense  of  a  360  degree  viewing  environment.  The  participants  within  our  virtual 
worlds  interact  with  the  virtual  objects  by  using  a  data  glove,  or  other  6  degree  of 
freedom  input  devices  such  as  the  Ascension  Bird,  the  Polhemus  Fastrak,  or  by  using 
a  3  degree  of  freedom  space  control  stick.  Furthermore,  we  are  exploring  less 
expensive  commercially  available  products  such  as  the  Mattel  power  glove,  the 
Logitech  Cyberman,  and  MIDI  keyboards  with  pitch  wheels,  data  sliders,  and  foot 
pedals,  as  alternative  low  cost  Input  devices  to  manipulate  virtual  objects.  Limited 
tactile  feedback  is  provided  by  a  Mondotronics  Tactor  and  limited  vibration  feedback 
by  the  Cyberman. 

Based  on  a  joint  effort  with  the  Human  Interface  Technology  Laboratory,  we 
have  established  a  comprehensive  research  program  to  investigate  the  relationship 
between  presence,  situational  awareness,  and  performance  within  virtual 
environments.  Questions  that  we  are  addressing  include  among  others:  How  do  we 
measure  the  level  of  presence  experienced  by  a  participant  within  a  virtual 
environment?  What  is  the  relationship  between  variables  such  as  frame  update  rate, 
display  resolution,  field  of  view,  and  presence?  Under  what  conditions  might  presence 
be  a  detriment  or  benefit  to  performance? 

We  are  not  limiting  our  studies  to  fully  immersive  virtual  environments,  but  are 
also  investigating  presence  for  augmented  reality  applications  and  for  real 
environments.  Thus  far,  the  greatest  emphasis  has  been  on  the  development  of 
subjective  metrics,  such  as  questionnaires  to  determine  the  degree  of  presence  one 
experiences  when  working  in  real,  augmented,  or  virtual  environments,  and  the 
development  of  performance  measures  to  evaluate  tasks  performed  within  virtual 
environments  such  as  manual  tracking,  or  spatial  navigation.  Our  work  on  presence 
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will  lead  to  a  validated  set  of  metrics  which  can  be  used  to  evaluate  presence  as  a 
function  of  the  characteristics  of  the  virtual  environment. 

Soatialized  Sound  Research 

Generally,  our  work  with  spatialized  sound  focuses  on  determining  how  best  to 
integrate  sound  into  the  virtual  environment.  Sound  has  two  properties  that  differ  from 
light.  Sound  is  intrusive  and  all-encompassing,  meaning  that  it  can't  easily  be  shut  out 
and  it  can  be  sensed  from  any  direction.  We  hope  to  increase  the  feeling  of  presence 
in  a  virtual  world  by  augmenting  the  visual  objects  in  the  world  with  specific  sounds 
emanating  from  the  same  apparent  location  in  the  three-dimensional  environment. 

The  use  of  nonspeech  sound  has  the  potential  to  add  a  great  deal  to  the  functionality 
of  computer  interfaces.  Sound  should  be  used  in  computers  as  it  is  in  the  world, 
where  it  conveys  information  about  the  nature  of  sound-producing  events.  Such  a 
strategy  leads  to  enhancements  to  interfaces  such  as  auditory  icons,  which  are 
everyday  sounds  meant  to  convey  information  about  computer  events  by  analogy  with 
everyday  events.  Auditory  icons  are  an  intuitively  accessible  way  to  use  sound  to 
provide  multidimensional,  organized  information  to  users. 

We  currently  have  the  capability  to  produce  "virtual"  three-dimensional  sound 
(or  spatialized  sound)  using  three  different  types  of  auditory  display  hardware.  The 
first  type  of  hardware  is  the  Focal  Point  card  produced  by  Bo  Gehring.  The  card  has 
the  capability  to  spatialize  two  sound  sources  simultaneously  and  independently.  The 
card  has  two  monophonic  inputs  and  one  stereoscopic  output.  Sounds  can  be 
spatialized  in  real-time  in  terms  of  azimuth,  elevation,  and  gain.  A  second  card  we  use 
to  spatialize  sound  is  made  by  Advanced  Gravis  Computer  Technology.  The  Gravis 
Ultrasound  card  can  preprocess  a  monophonic  sound  source  and  pls^  back  the 
spatialized  sound  in  real-time.  The  Gravis  Ultrasound  card  can  also  manipulate 
azimuth,  elevation,  and  gain.  Finally,  the  third  card  we  use  to  spatialize  sound  is  the 
Beachtron  card  made  by  Crystal  River  Engineering.  Unlike  the  Focal  Point  card  or  the 
Gravis  Ultrasound  card,  this  card  has  the  capability  to  accept  alternate  head  related 
transfer  functions  (HRTF)  besides  the  single  HRTF  that  comes  with  the  card.  In 
addition,  the  Beachtron  card  has  the  capability  to  spatialize  two  monophonic  sound 
sources  in  real-time  by  manipulating  azimuth,  elevation,  and  gain. 

For  spatialized  sound,  one  topic  of  investigation  concerns  the  accuracy  with 
which  a  listener  can  judge  the  position  of  a  sound  source  in  three-dimensions.  To 
investigate  this  topic,  we  recently  completed  a  study  in  which  we  compared  the  ability 
of  subjects  to  locate  objects  in  3-space  using  either  the  visual  or  auditory  modalities. 

In  general,  the  results  indicated,  not  surprisingly,  that  performance  using  the  visual 
modality  was  twice  as  accurate  (overall  about  7  degrees)  for  spatializing  stimuli  than 
the  auditory  modality  (overall  about  14  degrees).  However,  the  auditory  stimulus  was 
found  to  be  an  excellent  "pointer  for  the  eyes"  when  the  objects  were  located  outside 
the  subject's  visual  field  of  view.  In  another  study,  Jeff  Brandt  is  investigating  the 
human's  ability  to  localize  sound  sources  in  a  three-dimensional  acoustic  environment 
created  using  HRTFs  played  over  headphones  while  a  person  is  moving  within  that 
three-dimensional  environment.  A  Polhemus  Fastrak  is  used  to  track  a  person's 
location  and  head  orientation.  This  information  is  then  fed  to  a  sound  card  (Beachtron, 
Focal  Point,  or  Gravis)  that  imparts  the  localization  information  (similar  to  how  your 
head  and  pinnae  perform  the  same  process)  on  the  sound  before  it  reaches  your  ear 
canals.  The  result  is  that  objects  or  locations  in  a  three-dimensional  virtual  world  can 
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be  endowed  with  a  characteristic  voice  or  sound.  It  is  hoped  that  this  will  add  realism, 
aid  in  localization  and  identification,  and  increase  the  overall  feeling  of  presence  in 
virtual  worlds. 

Augmented  Reality 

Currently,  most  designers  of  virtual  worlds  start  with  a  visual  void,  then 
laboriously  model  scenes  with  polygons,  and  eventually  present  computer-generated 
stereoscopic  imagery  of  the  scenes  through  opaque  head-mounted  displays.  A 
problem  with  this  approach  is  that  most  current  workstations  do  not  have  sufficient 
computing  power  to  render  the  number  of  polygons  necessary  to  create  a  complex 
scene  at  a  high  enough  frame  rate.  This  often  leads  to  low  fidelity  virtual  environments 
in  which  it  is  difficult  for  the  user  to  suspend  disbelief.  In  addition,  there  are  many 
situations  in  which  we  would  like  to  interact  with  the  surrounding  real  word.  An 
augmented  reality  display  can  make  this  possible  by  presenting  imagery  that  enriches 
rather  than  replaces  the  real  world. 

Numerous  examples  of  monoscopic  real  world  imagery  integrated  into  graphics 
environments.  However,  the  real  world  has  a  wealth  of  depth  information  that  it  not 
available  using  just  a  monoscopic  display.  Our  approach  recognizes  this 
dimensionality,  therefore  over  the  last  year  we  have  developed  a  stereoscopic 
augmented  reality  system,  which  is  capable  of  integrating  stereoscopic  real-time  video 
with  stereoscopic  computer-generated  imagery  for  use  in  human  computer  interfaces. 

The  prototype  system  in  use  in  our  laboratory  was  built  by  Dav  Lion  and  Craig 
Rosenberg  and  can  be  broken  down  into  three  main  parts;  video  capture,  computer 
graphics  rendering,  and  signal  integration.  A  brief  overview  of  how  the  system  works 
follows.  The  video  capture  of  the  left  eye/right  eye  view  (both  necessary  for  creating  a 
stereo  image)  is  done  using  two  consumer  grade  S-VHS  cameras.  The  next  step  is  to 
digitize  the  two  images  using  two  live  motion  video  capture  boards.  The  computer 
graphics  is  generated  using  Silicon  Graphics  workstations.  Finally,  the  video  images 
and  computer-generated  graphics  are  merged  in  a  video  keyer  before  the  combined 
image  is  displayed  through  StereoGraphics  hardware  on  either  a  1 9  inch  monitor,  a  6 
foot  by  8  foot  rear  projection  screen,  or  an  HMD.  A  participant  within  an  augmented 
reality  world  is  viewing  a  stereoscopic  video  image  of  himself  overlaid  with 
stereoscopic  computer-generated  imagery  in  the  form  of  stars  circling  his  head.  A 
sensor  (Polhemus)  is  mounted  on  the  users  hat  to  track  head  position  and  is  yoked  to 
the  position  of  the  computer-generated  imagery.  Note  that  two  video  cameras  are 
used  to  capture  the  real  environment  necessary  to  create  a  stereo  image.  The  image 
on  the  SGI  monitor  appears  double  unless  viewed  using  StereoGraphics  shutter 
glasses.  Current  and  future  work  will  be  concentrated  on  improving  the  quality  of  both 
the  real-time  video  images  and  the  computer-generated  imagery,  together  with 
exploring  new  possible  application  areas  like;  computer  aided  instruction,  equipment 
maintenance  and  repair,  architectural  visualization,  and  telerobotics. 

Furthermore,  it  should  be  noted  that  there  are  significant  technical  problems, 
both  hardware  and  software,  which  must  be  solved  before  augmented  reality  systems 
are  more  widely  used.  One  of  the  significant  problems  is  the  placing  of  the  virtual 
image  exactly  where  it  should  be  in  the  real  world  scene.  For  applications  in 
medicine,  for  example,  this  will  be  critical.  However,  for  some  applications,  say 
entertainment,  exact  positioning  of  the  virtual  object  within  the  scene  may  not  be 
necessary.  For  purposes  of  demonstrating  the  feasibility  of  our  system,  we  solved  this 
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problem  essentially  by  trial-and-error  by  first  placing  the  transmitter  at  a  fixed  location 
and  then  adjusting  the  software  values  representing  the  position  of  the  virtual  object 
until  the  ring  of  stars  correctly  "floated"  around  the  person's  head.  For  our  system,  we 
also  had  to  solve  the  problem  of  correctly  yoking  the  movement  of  the  computer¬ 
generated  objects  (i.e.,  stars)  to  the  movement  of  the  user's  head.  This  was 
accomplished  by  using  a  head  positioning  sensor  and  by  placing  a  clipping  plane 
through  the  user’s  head  so  that  stars  which  floated  behind  the  person's  head  were  no 
longer  visible. 

Stereoscopic  Display  Design 

A  general  issue  in  the  design  of  spatial  displays  is  how  parameters  of 
perspective,  such  as  geometric  field  of  view  (like  zoom  in  film)  and  station  point 
distance  (like  dollying  in  film)  influence  perceptual  and  spatial  performance  using 
these  displays.  Manipulations  of  the  geometric  field  of  view  magnify  or  minify  the 
image,  while  manipulations  of  the  station  point  distance  result  in  compression  of  the 
depth  dimension  of  the  scene. 

Presenting  three-dimensional  information  using  traditional  two-dimensional 
displays  requires  mental  reconstruction  of  the  missing  dimension.  Three-dimensional 
displays  are  considered  a  more  intuitive  means  in  which  to  present  three-dimensional 
spatial  information.  However,  three-dimensional  displays  such  as  perspective  and 
stereoscopic,  still  required  a  certain  amount  of  mental  reconstruction  due  to  the 
perspective  distortions  which  are  inherently  part  of  perspective  displays.  The  degree 
to  which  distortion  influences  human  performance  is  dependent  on  the  perspective 
parameters  used  to  design  the  display.  Therefore,  it  is  critical  that  engineers  and 
designers  incorporate  appropriate  design  parameters  when  creating  spatial  displays 
to  minimize  the  effect  of  perceptual  biases  on  spatial  judgments. 

Claudia  Hendrix  and  Ove  Bjorneseth,  have  conducted  several  studies  to 
determine  what  display  conditions,  or  parameters  of  perspective,  used  in  designing  a 
perspective  display  will  minimize  human  errors  in  spatial  judgments.  Some  of  the 
design  parameters  they  have  investigated  include  the  effect  of  geometric  field  of  view, 
eyepoint  elevation  angle,  viewing  vector  angle,  target  location  in  the  vertical  and 
horizontal  dimensions,  stereoscopic  versus  monoscopic  display  conditions,  and  the 
interaction  of  these  factors  with  three-dimensional  sound  on  human  performance  in 
spatial  judgments. 

A  study  in  progress  is  designed  to  investigate  the  effect  of  the  presence  or 
absence  of  depth  cues  on  judgments  of  spatial  information.  The  experimental  set-up 
for  a  typical  study  generally  involves  a  subject,  wearing  time  multiplexed  shutter 
glasses  from  Stereographies,  viewing  a  stereoscopic  perspective  display  on  a  screen. 
The  subject  uses  a  screen  to  the  right  to  record  a  response.  This  study  will  allow  us  to 
determine  the  importance  of  individual  depth  cues,  the  effect  of  their  interaction,  as 
well  as  the  degree  to  which  they  influence  the  magnitude  of  error  in  spatial  judgments. 
The  accumulation  of  inforr>>ation  generated  by  the  perspective  display  studies  has 
resulted  in  human  factors  guidelines  that  can  be  used  to  design  perspective  displays 
so  as  to  maximize  human  performance  in  spatial  judgments. 
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Facilities 


Hardware 


Computer  Systems 

•  2  Silicon  Graphics  lncligo2  Extreme  Computer  Graphics  Workstation 

•  Silicon  Graphics  IRIS  4D  70/GT  Computer  Graphics  Workstation 

•  HP  340  Computer  Graphics  Workstation 

•  4  IBM  AT  Compatible  PC  (386/486) 

•  Texas  Instruments  Microexplorer 

•  Introl  Hard  Drive 

•  3  Macintosh  Computers 

•  Laser  Writer  II  Printer 

•  Okidata  Printer,  other  PC  printers 

Displays 

•  GE  Imager  610  Back-lit  6'x8'  Screen  Projection  System 

•  3  SGI  19”  High  Resolution  Stereo  Capable  Monitors 

•  Eye-Gen  3  Head-Mounted  Display 

•  Sony  PVM  2030  NTSC  Video  Monitor 

•  Virtual  Vision 

•  Stereographies  CrystalEyes  3D  Glasses 
Tranducers 

•  Ascension  Bird,  a  6df  position  and  orientation  tracking  device 

•  Polhemus  Fastrak,  a  6df  position  and  orientation  tracking  device 

•  Logitech  Cyberman,  6df  vibrotactile  mouse 

•  Tactor,  a  tactile  output  device 

•  Power  Glove,  4df  finger  positional  sensing  input  device 

•  Flight  Stick 

Audio 

•  3  MIDI  Interfaces  (Mac,  PC,  UNIX) 

•  Roland  Tone  Generator 

•  Ensoniq  Digital  Sampling  System 

•  12  Channel  Mackie  Audio  Mixer 

•  4  Channel  Realistic  Audio  Mixer 

•  Stereo  Amplifier 

•  Focal  Point  3D  audio  card 

•  Crystal  River  Beachtron  3D  audio  card 

•  Gravis  3D  audio  card 
« Infinity  Speakers 

•  Roland  MIDI  Keyboard 

Video 

•  Two  8mm  Video  Cameras  for  Stereo  Images,  with  Stereo  Mount  (RICOH) 

•  1  Canon  Camcorder 

•  2  Miniature  CCD  Cameras 

•  Two  Industrial  SVHS  Video  Tape  Recorders  (Panasonic  7500A) 

•  Two  Video  Capture  Cards  (New  Media) 

•  Video  Editing  Equipment  (Videonics) 
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•  Victor  NTSC  to  RGB  Converter 

•  Mediator  VGA  to  NTSC/SVHS  Converter 

•  Videologic  Scan  Converter 

•  Lyon  Lamb  Minivas  Animation  Controller 

•  ENC  VII  NTSC  Encoder/Sync  Generator 

•  Kitchen  Sink,  time  based  corrector 

•  Laird  Video  Keyer 

•  Video  Titler 

•  Pentax  35mm  Camera  with  Tripod  and  Filters 

•  35mm  Slide  Projector 

Simulator 

•  Automobile  Simulator,  full  scale,  fully  instrumented 

Software 

•  Alias  :  Software  for  modeling,  rendering,  and  animating  images 

•  I3DM  :  Software  for  modeling,  rendering,  and  animating  images 

•  Precept ;  Virtual  environment  development  software,  developed  in-house 

•  Open  Inventor  (SGI) 

•  Performer  (SGI) 
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